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ABSTRACT 

A datal>ased approach to vocational course assessment 
enables users to rank the quality of vocational education courses. 
Courses ranked highest may be commended. Courses ranxed lowest may be 
considered in need of improvement efforts. The Databased Course 
Assessment Hethod (DCAM) was develoi^d in the public domain and 
customized to the needs of the Cleveland City School District. The 
approach is designed to minimize the influence of implicit judgments 
and perceptions. The DCAH structure consists of three interrelated 
components: the information selection framework, the scoring process, 
and the ranking process. Forty-one courses were selected for the 
pilot test of the DCAM. A group of employers, school administrators, 
cind vocational administrators was convened to pilot test a procedure 
for obtaining DCAM information components and performance measure 
tfeights. One limitation in the application of the statistical 
procedures using pilot- test data is that these data were incomplet-^. 
Two conclusions can be drawn from the statistical analyses: (1) the 
DCAM approach can be statistically validated; and (2) the DCAM works 
optimally with stakeholder involvement. The data lend empirical 
support for stakeholders' involvement and show the relative 
contribution of the model's information components and performance 
measures. (Twenty-two esthibits are included. Appendix A contains the 
information set used in the pilot test. Appendix B contains the 
revised information set.) (NLA) 
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FOREWORD 



There continues to be a compelling need for database 
approaches to planning new vocational education courses 
and assessing ongoing ones to decide which of them are 
most in need of program improvements. Dr. Harold Starr 
and the Center on Education and Training for Employment, 
the Ohio State University, received funding from the 
Cleveland City School District to develop needed course 
assessment and planning methods that could be installed 
and implemr^ted by the district's Division of Vocational 
and Career ^^ducation. 

This report describes a database approach to 
vocational course assessment. The method enables users 
to rank the quality of vocational education courses. 
Courses ranked highest may be commended. Courses ranked 
lowest may be considered most in need of improvement 
efforts. 

Dr. Harold Starr directed efforts to develop the 
database course assessment method and customize it for 
use by the Cleveland Public Schools. He is the major 
author of this report. Dr. Starr is a Senior Research 
Specialist Emeritus, the Ohio State University and is a 
consultant to education agencies. 

Dr. Gary Grossman carried out the statistical 
analyses of pilot-test data and wrote the findings and 
conclusions of the analyses found in Chapter III of this 
report. He is a Research Specialist at the Center on 
Education and Training for Employment, the Ohio State 
university. 

The authors express their thanks and appreciation to 
staff of the school district's Division of Vocational 
and Career Education including casmira DiScipio 
(director), Steve Haiorca, John Perrin, and Richard 
Gore. These persons reviewed, contributed 
substantively, and critiqued components of the DCAM. 
Mr. Maiorca served as project monitor for the division 
and also did an outstanding job of collecting and 
organizing the pilot-test data. Thanks is also due Ann 
Holland who is on the staff of the district's Research 
and Analysis Department for her review and critique of 
the information set used in the pilot-test. 



Harold Starr 
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CHAPTER I 



INTRODUCTION 



The Federal District Court issued an order requiring 
the Cleveland City School District to improve the quality 
of job preparation (vocational) courses. The school 
system responded to the court by identifying a number of 
course improvement initiatives and initiating efforts 
leading to their implementation. 

One of the course improvement initiatives included 
implementing a more effective way to assess ongoing 
vocational courses and to plan for new courses. To this 
end, the district intends to implement a databased course 
assessment method that can be used to identify vocational 
courses most in need of improvements. It also intends to 
have available the kinds of information and data that are 
needed to plan for vocational education courses that will 
benefit both students and local employers. 

Local school administrators currently use many kinds 
of data for vocational course assessment and planning 
purposes. However, these data lack the kind of structure 
and customizing that are needed to compare vocational 
courses to determine which ones are most in need of 
improvements and to plan new courses. 

staff within the district's Division of Vocational and 
Career Education conducted efforts to locate tested 
databased vocational course assessment methods. None were 
found. A Request for Proposal to design and pilot-test a 
databased course assessment method and formulate needed 
course planning data was then prepared by the school 
district and sent to prospective bidders. The school 
district selected The Center for Education and Training 
for Employment, the Ohio state University and Dr. Harold 
Starr to conduct a scope of work to achieve these 
objectives. 

The present report describes the development and 
pilot-test of the databased course assessment method 
(DCAM). Data useful for planning new vocational courses 
is found in a separate report. The latter report includes 
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Infomatlon and data obtained from doctiments describing 
local labor market conditions and needs and from a 
synthesis of interviews conducted with local business and 
labor persons. 



Contents Qt This Rer^ 

This report contains two chapters besides the current 
one. Chapter ZX, The Omtabased Course Assessment Method 
(DCAM), consists of three sections. The first section 
includes a brief discussion of the common and unique 
characteristics of the DCAM. The second section describes 
the structure of the DCAM. The third section lists tasks 
local users nay find helpful as they customize, install, 
and operate the DCAM. Chapter III, The Pilot-Test of the 
DCAM, includes a summary of the developmental activities 
leading to the pilot-test of the DCAM. It also includes a 
description of activities and outcomes of the pilot-test. 

A number of exhibits and appendixes are found in the 
text of the present report. These materials are intended 
to help the reader better understand the DCAM and the 
steps required to customize, install and operate it. 



CHAPTER II 



THE DATABASED COURSE ASSESSMENT METHOD (DCAM) 



Vocational course assessment methods are mainly 
qualitative in nature. They rely mainly on expert 
judgments, observations, and intuition by vocational 
educators and other stakeholders (e.g., employers, 
parents) to produce reliable and valid assessment 
outcomes . 

It is sometimes difficult to interpret the outcomes of 
qualitatively based course assessment methods. It is not 
unusual to find evaluators differing in their findings. 
Reasons for differences may include the fact that 
evaluators may look at different course behaviors and 
attributes or look at the same behaviors and attributes 
differently. 

An alternative to qualitative course assessment was 
formulated by the author of this report. It was designed 
as a databased course assessment method (DCAM). One of 
the intentions of the design of the DCAM was to minimize 
the influence of implicit judgements and perceptions that 

are a part of qualitative assessment methods.^ 

Federal vocational education legislation since 1963 
has promoted the application of the best data available 
for reporting and accountability purposes. Many state and 
local vocational education agencies have responded by 
developing more sophisticated management information 
systems. The presence of these systems was recognized in 
the conceptual development of the DCAM. 



iThe remainder of this section draws heavily on material 
contained in Harold Starr. Increasing Vocational 
Education Program Relevance: A Databased Approach. 
Columbus: The National Center for Research In vocational 
Education, The Ohio State University, 1987. 
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The (DCAM) is a conceptual product. It is the product 
of research and developnent efforts carried out by the 
National Center for Research in Vocational Education (now 
the Center for Education and Training for Enployment) , The 
Ohio State University. 2-5 To date, the DCAM has not been 
fully tested and installed in any school system. 

Funding for the research and development described in 
the publications listed below cane from three public 
soui.es. Funding for publications one, three, and five 
came from the U. S. Department of Education. Funding for 
publication two came from the Office of Instructional 
Programs, Georgia Department of Education. Funding for 
publication four came from the Vocational and industrial 
Training Board, the Republic of Singapore. 

The DCAM as a conceptual product was developed mainly 
in the public domain. The purpose of the present project 
is to customize it to the needs of the Cleveland City 
School District. 



^Harold Starr, The Evaluation Index- Columbus: The 
National Center for Research in Vocational Education, The 
Ohio State University, 1988. 

^Harold Starr. Increasing Vocational Education Program 
Relevance: A Databased Approach. Columbus: The National 
Center for Research In Vocational Education, The Ohio 
State University, 1987. 

^Harold Starr. The Development of a Practical Model for 
Planning Vocational Training. In Management Information 
System for Vocational Education and Training — Final 
Report (ASEAN-Australian Development Education Project — 
Educational Management Information system). Singapore, 
Republic of Singapore: The Vocational and Industrial 
Training Board, 1984. 

^Harold Starr, Harold, Merz, and Gale. Zahniser, Using 
Labor Market Information for Vocational Planning. 
Columbus: The National Center for Research in Vocational 
Education, The Ohio State University, 1982. 
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Among the characteristics that distinguish the present 
assessment method, the DCAM, from traditional ones are the 
following: 

o The DCAH is databased. 

o Comparisons of dissimilar course 
performances are made possible by a 
data-normalizing technique. 

o The relative contribution (i.e., the 
relative weight) of different kinds of 
course behaviors or attributes in the 
assessment process can be obtained by 
having stakeholders value them or by 
using statistical procedures. 

o The DC AM enables users to make 
comparisons between the same or 
different kinds of vocational courses. 
For example, users can compare all or a 
subset of ongoing vocational courses, 
the same vocational courses offered at 
different locations, or vocational 
courses within or between career 
clusters . 

o Use of a microcomputer with spreadsheet 
software may enhance the speed of 
estetblishing rankings of courses for 
overall performance outcomes. 

Two additional features of the DCAM are »-'orth noting. 
First, school administrators will be able to conduct 
'*desk-top audits** of DCAH outcomes to decide which 
vocational courses should be given priority attention as 
candidates for program improvements and which ones should 
be commended. Second, the DCAM complementi:; rather than 
replaces traditional methods of assessing vocational 
education courses (e.g., PRIDE, supervisory judgments). 
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The DCAM Structure 



The DCAM structure consists of three interrelated 
components: (X) the Information Selection Fre -^iWorK, (2) 
the Scoring Process, and (3) the Ranking process. These 
three components are described next. 

THE INFORMATION SELECTION FRAMEWORK 

Users of the DCAM will need to find out what specific 
quantifiable performance information to include for course 
assessment. The Information Selection Framework provides 
a handy way to help DCAM users to achieve this task. The 
Information Selection Framework consists of five related 
elements. These elements are as follows: 

o Information components 

o Information categories 

o Performance indicators 

o Performance measures 

o Performance measure outcomes and scores 
The four elements are described below. 

Xnf.ftrmatiQn Cgmppnents 

vocational educators typically are concerned with five 
kinds of information when assessing vocational education 
courses. These five kinds of information are as follows: 

1. Input/Context 

The input/context component includes 
information about student demographics 
and employment conditions that are 
likely to effect course performance or 
the relevance of vocational offerings to 
employment needs of students and 
employers. 
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2. Instructional Processes 



The Instruct iona J. processes c&nponent 
includes infonsation about instructional 
and support practices and conditions 
that are likely to influence course 
outputs, outcomes, or benefits. 

3. student Outputs 

The student outputs component includes 
information about the extent to which 
students complete a designated grade or 
course of instruction. 

4. vocational Course Outcomes 

The vocational course outcomes 
component includes information abort 
skill levels and educational 
achievements of course completers and 
their success in finding employment and 
pursuing further education. 

5 Vocational Course Benefits 

The vocational course benefits 
compor.'^nt includes information about the 
economic and social benefits accruing to 
the individual, the economy, and to 
society when students get vocational 
instruction (e.g., wages, taxes paid 
locally) 

Information Categories 

The information components element of the Information 
Selection Framework serve as an organizing scheme for 
formulating specific categories of information. For 
example, if instructional processes is chosen as an 
information component, then course popularity and course 
cQsta may serve as information categories under 
instructional processes. Exhibit l lists information 
categories used in the pilot-test of the DCAM in the 
Cleveland City School District. 



Exhibit 1 - List Of Information Categories By Information Component 

CONTEXT/INPUT COMPONENT 

o Enroliinent Equity 
o Course Popularity 

PROCESSES COMPONENT 

o Cour^Ct^te 

o Private Sector Support 

0 Secondary-Postsecondary Articulation 

o Professional Development Experiences 

o Instructional Design 

0 Student Organization Participation 

OUTPUTS COMPONENT 

0 Course Attrition/Completion 

OUTCOMES COMPONENT 

o Job/education Status of Completers 
o Professional Recognition of Completers 

BENEFITS COMPONENT 
o Wages of Completes 
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Performance Indicators 



One or nore performance indicators (performance 
criteria) need to be formulated for each information 
category in the Information Selection Framework. If 
course popularity is chosen as a performance category, 
then the extent that students enroll in a vocational 
course of their choice could serve as a performance 
indicator. Exhibit 2 lists performance indicators used in 
the pilot-test of the DCAM in the Cleveland City School 
District. 

Performance Measures 

One or more performance measures must be formulated 
for each performance indicator. For example, if the 
extent that students enroll in a vocational course of 
their choice serves as a performance indicator, the 
percent of first-year students in a vocational course 
selecting it as their first choice might serve as a 
performance measure. Exhibit 3 lists performance measures 
used in the pilot-test of the DCAM in the Cleveland City 
School District. 

Performance Measure Outcomes 
And Scores 

Lastly, a set of performance measure outcomes and 
scores is formulated for each performance measure. Outcome 
measures take the form of numbers, ratios, trends, 
discriminating values, or ranks. 

The following might be a set of performance measure 
outcome statements for the performance measure the percent 
of first-year students in a vocational course selecting it 
as their first choice: 



This course Js among the three with the greatest percent of 
first-year students selecting it as their first choice. 

Ff lis rourse ranks above the median for the percent of first- 
year students selecting it as their first choice. 

This course ranks at or below the median for the percent of 
first-year students selecting It as their first choice. 

This course Is among the three with the smallest percent of 
first-year students selecting it as their first choice. 

Performance data are not available. 
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Exhibit 2 - List Of Performance Indicators • 
CONTEXT/INPUT COMPONENT 

o Enroilment Equity 

The extent that there Is equity in enrollment of Black 
students in vocatimal cf^jrses 

T/» extent of sex equity in enrollments in vocational courses 
o Course Popularity 

The extent that students enroll in a vocational course of 
their cht^ce 

The extent that vocational courses make use of training 
cecity 

The extent that first-year vocational students return to the 
same course for a second year of instruction 

PROCESSES COMPONENT 

o Course Costs 

Costs associated with operating vocational courses 

o Private Sector Support 

The extent that private sector sources contribute to the 
opera^ons of vocationsJ courses 

The extent of female and minority participation on 
vocational ccHirse advisory committees 

o Secondary-Postsecondary Articulation 

The extent of secondary and postsecondary articulation 
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0 Professionai Development Experiencesi 

The extent that instructors participate in professionai 
develofmertt experiences 

o instructional Design 

The extent tfmt vocation^ education course curricula are 
c&npetency-baaed according to state department of 
education stwTdmds 

0 Student C^ganization Participation 

The extent of participation by stixients in vocational 
edur^tirm stixient orgmiizations 

OUTPUTS COMPONEIMT 

o Course Attrition/Completion 

Ttm extent of student attrition from vocation^ education 
courses 

The extent that sWdents compiete their vocational 
instructiw 



OUTCOMES COMPONENT 

o Job/education Status of Completes 

The extent of training-related job placement assistance tc 
vocatiwi^ course completers 

The extent that vocations^ course cc^pleters succeed in 
finding jc^ or furthering their educatiw 
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o Professional Recognition 



ThB extent that vocational course cmpleters get licensing 
or certification (when applicable) 

BENEFITS COMPONENT 

o Wages of Completers 

Entry-level wages of vocational course completers 

^Perfomiance Indicators are In Italics 
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Exhibit 3 - List Of Performance Measures! 

CX)NTEXT/!NPUT COMPONENT 
Enrollment Equity 

The extent that there is equity In enrollment of Black students In vocational courses 

Th9 percent dsvlatlon from the district's goal of 70 percent 
for Black ftrat-year opening enrollment in each course 

The extent of sex equity in enrollments in vocational courses 

The percent deviation from the go^ of 50 fxrcent female for 
first-year opening enrollment in each courses 

Course Popularity 

The extent that students enroll In a vocational course of their choice 

The percent of first-year students in a vocational course 
selecting it as tfieir first choice 

The extent that vocational courses make use of training capacity 

The percent of first-year opening vocational course 
enrollment to first-year course capacity 

The extent tfwl first-year vocational students return to the same course for a second year 
of instruction 

The percent of first-year vocational students who return to 
the same course for a second year of instruction 



PROCESSES COMPONENT 
Course Costs 

Costs associated with operating vocational courses 

The per-student operating costs for vocatloneU courses 
The per-completer operating costs for vocational course^ 
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Private Sector Support 

The extent that privatfc sector sources contribute to the op^tlons of vocational courses 

The presence of tangible support tNs year and last year by 
private secfw scmcea 

The extent of female and minority participation on vocational course advisory committees 

The mm^r of persons frwn each sex and the number of 
minority persons represented on vocational course advisory 
carmittees 

Secondary-Postsecondary Articulation 

The extent of secondary and postsecondary articulation 

A secondary-postsecondary articulation agreement is in 
force or Is in the works 

Profei^lonai Development Experiences 

The extent that Instructors participate in professional development experiences 

Whether vocational instructors participated in proiesslonai 
develofment experiences this year and last year 

Instructional Design 

The extent thiW vocatfonal education course curricula are competency-iaased according to 
state department of education standards 

Competency'tximd vocational curriculum is in force or is in 
the works 

The extent that vocational courses use computer ^jftware for skill enhancement or 
remedial education purposes 

The nurr^r of years f/wf vocational courses use computer 
software for skill enfwicement and remedial education 
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Student (^^nization Participation 



The extent of participation by students in vocational education student organizations 

Tfw perc&nt of stixlents in each v(xatiwial course 
participating in a vocaU<mal student organization 

OUTPUTS COMPONENT 
Course Attrition/Completion 

The extent of student attrition from vocational education courses 

The percent of stixtents dropping out from vocational 
courses 

The extent that students complete their vocational instruction 

The percent of course completers to first-year course 
capacity 

OUTCOMES COMPONENT 
Job/education Status of Completers 

The extent of tralning-relatPd job placement assistance to vocational course completers 

The percent of comf^eters placed in training-related jobs 
with help from scho(^ staff 

Tlie extent that vocational course completers succeed in finding jobs or furthering their 
education 

The percent of vocation^ course completers currently in 
training-reiated jobs, in the military, or pursuing further 
educaticm 




Professional Recognition 

The extent that vccatlonal course completers get licensing or certification (when 
applicable) 

The percent of vocational course completers who get 
licensed or certified 

BENEFITS COMPONENT 
Wages of Completers 

Entry-level wages of vocational course completers 

The median entry-level wages earned by vocational course 
completers getting training-related fobs 

^Performance Measures are In italics 
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The way in which noncalized performance measure outcome 
scores are assigned to performance measure outcome 
statements is described in the next section of this 
report. 

There are no specific rules for establishing the best 
or optimum number of performance measures and performance 
measiire outcome statements for performance indicators. It 
should be easier to identify appropriate and useable ones 
where an education agency has a well-develcped management 
information system and when users gain experience in 
operating the DCAM. This experience can be of help in 
assessing the credibility and stability of various 
performance measures and performance measure outcomes. 

Exhibit 4 depicts the relationship between an 
Information component and category, a performance 
indicator, a performance measure, and performance measure 
outcomes . 

THE SCORING PROCESS 

Normal jgina Perfermanee 
Measure Outcomes 

Each performance measure in the Information Selection 
Framework has a set of performance measure outcomes 
associated with it. In addition, the framework requires 
that each performance measure should have the same number 
of performance measure outcome statements. All of the 
performance measure outcome statements comprising a 
performance measure are then assigned a simple range of 
scores (e.g., one to four or one to five). 

The performance measure outcome scores are considered 
normalized because a performance measure outcome score of 
five (or four or three, etc.) in one set of performance 
measure outcome statements is equivalent to a performance 
measure outcome score of five (or four or three, etc.) in 
any other set of performance measure outcome statements. 
Exhibit 5 contains two sets of performance measure outcome 
statements that are scored five, four, three, two, and 
one. 
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Exhibit 4 - An Informptlon Component. Category. Performance Indicator. 
Performance Measure. And Perf(MTnance Measure Outcomes. 

PROCESS COMPONENT 
Information Category: Course Costs 

Performance indicator: 

Costs associated with operating vocational courses 
Performance Measure 

The per-student operating costs for vocational courses ' 

Porformanco Moasura Outcomas 

This course ranks first In its career cluster for lowest per- 
student operating cost. 

'nils course ranks above the median in its career cluster (but 
not first) for per-student operating cost. 

This course ranks at or below the median in its career cluster 
(but not last) for per-student operating cost. 

Jh\s course ranks last In Its career cluster for highest per- 
student operating cost. 

Performance data are not available. 
^The calculation of per-student operating cost does not include state reimbursement. 
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Exhibit 5 - Performance Measure Outcomes and Scores 



Peif ormance Measure Outcomes and Scores (were 5 Is best performance) 

|51 This course ranks first In Its career cluster for lowest per- 
student operating cost. 

[4 J This cowse ranks above the median In its career cluster for 
pi^-student operating cost. 

{31 This course ranks at or below the median in Its career cluster 
tor psr-student operating cost. 

12] This course ranks last In Its career cluster for highest per- 
student operating cost. 

[ 1 ] Perfomianco data are not available. 

Performance Measure Outcomira and Scores (where 5 is best performance) 

[5] "nils course Is among the three with the greatest percent of 
first-year students selecting it as their first choice. 

(41 This course ranks above the median for the percem of first- 
year students selecting it as their first choice. 

(31 This course ranks at or l^iow the median for the percent of 
first-year students selecting it as their first choice. 

(21 This course is among the three with the smallest percent of 
first-year students selecting It as their first choice. 

( 1 1 Performance data are not available. 
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Two dissimilar Hinds of data presented in exhibit 5 
may be compared in the following way. If a vocational 
course ranks at or below the median for the percent o£ 
first-year students selecting it as their first choice, it 
gets a score of three. It also gets a score of three if 
it has a per-student operating cost that is at or below 
the s^ian in its career cluster. The normalizing process 
makes the former performance measure outcome equal to the 
latter one even though they are substantively dissimilar. 

Another example of using normalized scores is as 
follows. If course "A" ranks above the median for the 
percent of first'-year students selecting it as their first 
choice.it receives a score of four. If course "A** has the 
lowest per-student operatii^ cost in its career cluster it 
receives a score of five, course "A" receives a total 
score of nine for its performance on these two measures. 
If Course "B" ranks at or below the median for the percent 
of first-year students selecting it as their first 
choice.it receives a score of three. If course "B" has 
the highest per-student operating cost in its career 
cluster it receives a score of two. Course "B" receives a 
total score of 5 points for its performance on these two 
measures. Therefore, course "A" performs more adequately 
than course "B". 

Weighting Information Components 
And PerfQ pnance Measures 

Using Stakehold ers To Obtain Weights 

Persons involved in applying qualitative methods to 
assess vocational education courses often give more weight 
(i.e., value) '-o some kinds of information than to others. 
Course assessors also frequently value some kinds of 
performances measures more than others. It is also likely 
that this valuing or weighting differs among assessors. 
The valuing process is usually implicit in nature and its 
impact on course assessment is usually either not taken 
into consideration or is unknown. 

The DCAM recognizes the existence of this phenomenon 
and has incorporated procedures to make the valuing 
process explicit and objective. A suinmary of the 
procedures for weighting information components and 
performance measures is as follow:^; 

o The assessors or a group of 

stakeholders is selected and convened to 
weight information components. 
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o A method is established to reconcile 
differences among assessors (i.e., reach 
a consensus) about what weights to 
assign to the information components. 

o Instructions are given that 100 points 
are to be divided among the information 
components. If there are five 
components and all information 
components are perceived as equal in 
importance ( an unlikely scenario) than 
each of them receives twenty points, 
otherwise, each component receives a 
different weight of importance but the 
total for all components must equal lOO 
points . 

o Stakeholders weigh each of the 

information components. The weights are 
shared with the group and the process of 
reaching consensus is put in place. 
What results are weights for information 
components that are explicit in nature. 

Performance measures could be weighted by the 
assessors or stakeholders in three ways. First, the 
persons doing the weighting could decide that all 
performance measures within an information component will 
be judged as equal in importance. Here, the number of 
performance measures in each information component is 
divided into the weight for its information component. If 
instructional Processes is assigned a weight of thirty- 
five points and it has five performance measures under it, 
each of them receives a weight of seven points. 

Second, each performance measure within an information 
component may be assigned its own weight. In this case, 
the sum of the performance measure weights must equal the 
weight assigned to the information component. Assume that 
an information component received a weight of thirty-five 
points and their are five performance measures. Each of 
the five measures may be assigned a different weight but 
the total of the five weights must equal thirty-five 
points. 

Third, the process of weighting information components 
is dropped. The entire set of performance measures are 
weighted and they must total 100 points. 
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A group of stakeholders was convened to decide on 
weights to be used in the pilot-test of the DCAM. The 
outcomes of the convening are described in Chapter III. 

Using Statistical Procedures 
To Obtain Weights 

Regression analysis techniques can also be used to 
establish weights of importance for information components 
and performance measures. The Request for Proposal 
stipulated that the contractor use regression analyses to 
arrive at weights. The regression analyses used to 
perform the weighting tasX and their outcomes are 
described in Chapter III. 



THE RANKING PROCESS 



Courses bei~ig assessed need to be ranked to determine 
which ones are most in need of improvements and which ones 
should be commended. This task can be accomplished 
efficiently by using a ranking matrix. 

The ranking matrix may be generated manually or by using a 
computer spreadsheet program. Users may find it more 
efficient to use the latter. Floppy disks containing 
ranking matrixes used in the pilot-test of the DCAM in the 
Cleveland City School District have been supplied to the 
division. 

Seven tasks should be carried out to produce a ranking 
matrix. These tasks are as follows: 



o Information components and their 
associated performance measures are 
entered as labels in matrix columns (see 
exhibit 6). 

o The names of the courses being assessed 
are entered as labels in the matrix rows 
(see exhibit 7). 

o Weights of importance are as'^igned by 
stakeholders to the information 
components and the performance measures 
associated with them (see exhibit 8). 

o Normalized performance measure outcosie 
scores are entered in the cells 
corresponding to their course and 
performance measure intersects (see 
exhibit 9). 
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The normalized outcome scores are 
recalculated to reflect the weight given 
to their s>erformance measures (see 
exhibit 10). 

The weighted normalized scores within 
each matrix row are summed to obtain a 
total performance score for each 
vocational course (see exhibit 11). 

The vocational courses are then ranked 
using the performance score totals. The 
highest ranked courses are those to be 
commended. The lowest ranked courses 
become candidates for program 
improvements (see exhibit 12). 




Exhibit 6 - Information Components And 
Performance Measures 

Context ProPMSOT 

Weight % 

PrfmMsrs ABODE F Q 



Exhibit 7-IJst Of Courses In The Ranking Matrix 

Context Prooesset 

Weight % 

PrfmMsre ABODE F G 

COURSES 
ALPHA 

BETA 

GAMMA 

DELTA 
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Output Outeorms Bnft Total 

OP Q R S T 



Output Outoomss Bnft Total 



OP O R S T 



ERIC 



31 

25 



Exhibit 8 - Weighted Information Components 
And Performance Measures 



Conttxt^put 

Weight % igiijij* ^ 

PrtmMsre A S C 0 6 

WtSoor* 2.7 2.5 S.4 J.9 5.5 



F G H 
2.5 2.3 3.0 



I J K L 
1.9 2.6 40 4.1 



COURSES 
ALPHA 

BETA 

GAMMA 

DELTA 

Exhibit 9 - Normalized Performance Measure 
Outcome Scores 



Weight % 
PrtmMire 
Wt Score 2.7 



Context/Input 

20 



A B 



2.5 



C 

5.4 



D E 

3.9 5.5 



Processes 

F G 

2.5 2.3 



H I J K L 
3 19 26 4 41 



COURSES PrfmMsre A 8 C D E 



ALPHA Norm Sc»fB 13 4 0 3 3 



F G H 1 J K L 
"4" '4 3 2" i' 3 1' 



SETA 



GAMMA 



DaTA 



Norm Score! 33041 1144231 



Norm Score! 43041 2243341 



Norm Score 



33012 1143341 



ERIC 



25 



9 ^ 



Om^Mit* OutoonHM Bwift Totml 

/O 30 /S l^t> 

•mm mm 

MN OP QRS T 

2.1 2.4 4.3 S.7 10.2 1^9 6.0 15.0 'ZMik^A 

''H9 



Outputs Outewnss Benft Total 

MN OP QRS T 

2.1 2.4 4.3 5.7 0.2 129 6.9 15.0 ;x!$»Siiy.: 



M N OP 0 R S 

^ - ^ 

1 4 2 4 0 4 0 

0 4 1 2 0 2 0 

0 4 1 1 0 4 0 
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Exhibit 10 ~ Recalculated Performance Measure 
Outcome Scores 



CofitBxI/tnput 

PrImMw A B 
Wt Seor» 2.7 2.5 



C 
5.4 



D 

3.9 



S.S 



P fOBBS MS 



F 

2.5 



G H 

23 3.0 



J 

1.9 



J 

2.6 



K 

4.0 



L 

4.1 



COURSES 
ALPHA 

eeA 

GAMMA 
DELTA 



Prfm Msra 

H»mSeot9 
NSxWS 

Norm Scots 
NSxWS 



B 



H 



3.0 4.0 0.0 3.0 3.0 



3.0 
8.1 



Norm Scots | 4.0 
NSxWS i 10.8 

j 

Norm Soors | 3.0 
NSxWS 



4.0 4.0 3.0 2.0 10 3 0 



3.0 
7.S 

3.0 
7.5 

3.0 
7.5 



0.0 
0.0 

00 
0.0 

0.0 
0.0 



4.0 
15.6 

4.0 
15.6 

1.0 
3.9 



1.0 
5.5 

1.0 
S.S 

2.0 

n.o 



1.0 
2.S 

2.0 
5.0 

1.0 
2.S 



1.0 
2.3 

2.0 
4.6 



4.0 4.0 

Ik.O 7.6 

4.0 3.0 

12,0 5.7 



1.0 4.0 30 
2.3 12.0 5.7 



2.0 
5.2 



1.0 



8.1 10.0 0.0 11.7 16.5 10.0 9.2 9.0 3.8 2.8 12.0 4.1 



3.0 1.0 
12.0 4.1 



3.0 4.0 
7.8 16.0 



1.0 
4.1 



3.0 4.0 10 
7.8 16.0 4.1 



ExhIbH 1 1 ~ Weighted Sums Of Scores In 
The Ranking Matrix 



Cootexi/input 

,2o 

PrfmMsr© A B 
Wt Scots 2.7 2.5 



C 
5.4 



D 

3.9 



E 

5.5 



Processss 

F Q 
2.5 2.3 



H 
3 



] 

1.9 



J 

2.6 



K 
4 



L 

4.1 



COURSES 
ALPHA 

BETA 

GAMMA 

DELTA 



Pffm Msrs 



B 



r«»TnScore| 3.0 



NSxWS 



8.1 



ritormScfM ! 3.0 



NSxWS 



Norm Scots 1 4.0 



8.1 



NSxWS 



108 



Norm Scots ; 3 0 



NSxWS 



8.1 



4.0 
10.0 

3.0 
7.5 

3.0 
7.5 

30 
7.5 



0.0 
00 

0.0 

ao 
oo 

0,0 

0.0 
00 



3.0 30 
11.7 165 



4.0 
15.6 

4.0 
15.6 

1.0 
39 



1.0 
5.5 

I. 0 
5.5 

2.0 

II. 0 



4.0 
100 

1.0 
Z5 

2.0 
S.0 

1.0 
2.5 



4.0 
9.2 



2.0 
4.8 

1.0 
2.3 



H 

"3.0" 
9.0 



2.0 
3.B 



10 4.0 4.0 

2.3 12.0 7.8 



4.0 3.0 

1^0 5.7 

4.0 3.0 

12.0 5.7 



10 
25 

2.0 
5.2 

3,0 
7.8 

30 

7.8 



30 
12.0 

3.0 
12.0 

4.0 
16.0 

4,0 
160 



10 
4.1 

1.0 
4.1 

1.0 
4.1 

1.0 
4.1 
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BEST COPY AVAILM 



Outputs Outcomes 8mft Total 

/O 3o /S- /0C> 

MN OP ORS T 

2.1 2.4 4.3 5.7 10.2 12.9 6.9 15.0 ii'iiiii'jiSf^S ^ 



M N 

2.1 9.6 12.9 11.4 0.0 25.8 0.0 1S.0 | 

{ 
t 

1.0 4.0 ZQ 4.0 0.0 4.0 0.0 0.0 | 

2.1 9.6 B.6 224) 0.0 51.6 0 0 0.0 f 



0.0 4.0 1.0 2.0 0.0 2.0 00 4 0 { 
0.0 9.6 4.3 11.4 0.0 25.8 0.0 60.0 1 



00 4.0 .0 1.0 0.0 4.0 0 0 0 0 I 
0.0 9.6 4.3 5.7 0.0 51.6 0.0 0.0 | 



M N 

2.1 2.4 



Outputs 

0 P 
4.3 5.7 



so 

Q R S 
10.2 12.9 6.9 



Bsnft Total 
T 

15.0 iWfiii^ 



M 



N 



TOTAL 



1.0 4.0 3 0 2.0 0.0 2.0 0.0 1.0 \ 44.0 

2.1 9.6 12.9 11.4 0.0 25.8 0.0 15.0 | 173.8 

1.0 4.0 2.0 4.0 0.0 4.0 0.0 0.0 { 42.0 

2.1 9.6 86 2Z8 0.0 51.6 0.0 00 | 177.1 

00 4.0 10 2.0 0.0 2.0 0 0 4.0 I 44.0 

0.0 9.6 4.3 11.4 0.0 25.8 0.0 60.0 \ 205.7 



0.0 4.0 
0.0 9.6 



1.0 1.0 
4.3 5.7 



0.0 4.0 0.0 
0.0 51.6 00 



0.0 i 

0.0 5 



36.0 
152-1 
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Exhibit 1 2 - Ranking Courses 



ConMxt /Input 

2o 

W«^M% |::'>:;:3!(): 

PrfmMsre A BCD 

WtSoor* Z7 2.5 5.4 39 



E 

5.5 



Prooes»s 

mm 

F G H 
2.5 2.3 30 



1.9 2.6 4,0 



L 

4.1 



COURSES 
ALPHA 

BETA 

GAMMA 

DELTA 



Prfm Msre 

N«m Score 
NSxWS 



B 



30 4.0 
B.I 10.0 



0.0 
00 



30 
11.7 



3.0 4.0 4.0 
18.5 



^to^m8c50f•i 3.0 3.0 0.0 4.0 10 



NSxWS 



NormSoore j 
NSxWS I 

i 

Norm Scons | 
NSxWS i 



8.1 

4.0 
10.8 

3.0 
81 



7.5 0.0 158 5.5 



30 

7.5 

3.0 

7.5 



00 

0.0 

0.0 
0.0 



4.0 
15.6 

10 

3.9 



1.0 
5.5 

2.0 
11,0 



10.0 92 



1.0 
2,5 

2.0 
5.0 

10 
2.5 



10 
Z3 



H 

9.0 

4,0 
12,0 



2.0 ».0 

4.6 '2.0 

1.0 4.0 

2.3 i.:.0 



1 J K 

"2'o i"0~ 30 
3,8 2.6 12.0 



40 

76 

3.0 
57 

3,0 
57 



2.0 
52 



3.0 
12,0 



30 4.0 

7.8 1B.0 

3.0 4.0 

7.8 16.0 



1.0 
4.1 

1.0 
4.1 

1.0 
4.1 

1.0 
41 
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30 



M 
2.1 



N 

2.4 



Outputs 
/O 

0 P 
4.3 5.7 



M N 



1.0 
21 

Q.0 
00 



4.0 
9.6 

4.0 
9.6 



OutCOfTWS 

...So 

*0'* R 
10.2 12.9 



1.0 4.0 3.0 2.0 

2.1 9.6 12.9 11.4 



2.0 4.0 

8.6 22.8 

1.0 2.0 

4.3 11.4 



S 

8.9 



0.0 2.0 0.0 

0.0 25.8 ao 

0.0 4.0 0.0 

0.0 51.6 0.0 

0.0 2.0 0.0 

0.0 25.8 00 



Banft 

T 

15.0 



"i:6] 

15.0| 

\ 
j 

0.01 

0.01 

\ 
\ 

4.0l 
60.01 



Total 



99' 9 



TOTAL 

44.0 
173.8 

42.0 
177.1 

44.0 
205.7 



iRANK COURSE SCORE ! 



1 
2 
3 
4 



GAMMA 
BETA 
ALPHA 
DELTA 



205.7 i 
177.1 
1738 
152.1 



00 4.0 
0.0 9.6 



1.0 1.0 
4.3 5.7 



0.0 4.0 0.0 OOi MO 
0.0 51.6 00 O.oi 152.1 
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yajor Tasks To Install And Operate The DCAM 



There are 14 major tasks to be performed when 
installing, operating, and recycling the DCAM. These 
major tasks are as follows: 



THE INFORMATION FRAMEWORK 

1 UST INFORMATION COMPONENTS THAT WILL BE USED WITH 
THcDCAM. 

2. SPECIFY INFORMATION CATEGORIES FOR EACH ASSESSMENT 
COMPONENT. 

3. ASSIGN ONE OR MORE PERFORMANCE INDICATORS TO EACH 
!r>FORMATION CATEGORY. 

4. SELECT ONE OR MORE PERFORMANCE MEASURES FOR EACH 
INFORMATION CATEGORY. 



THE SCORING PROCESS 

1. IDENTIFY THE SCORING PROCESS TO BE USED FOR 
NORMALIZING PERFORMANCE MEASURE OUTCOMES. 

2. FORMULATE PERFORMANCE MEASURE OUTCOME STATEMENTS THAT 
ARE COMPATIBLE WITH THE NORMAUZING PROCEDURE. 



THE RANKING PROCESS 

1. CONSTRUCT A RANKING MATRIX. 

A. LIST COURSES BEING.ASSESSED IN MATRIX ROWS. 

B. LIST INFORMATION COMPONENTS Alsffi THEIR ASSOCIATED 
PERFORMANCE MEASURES IN MATRIX COLUMNS, 

C. USE COLUMN ROW INTERSECTS TO RECORD NORMALIZED 
PERFORMANCE MEASURE OUTCOME SCORES. 



2. ASSIGN NORMALIZED PERFORMANCE MEASURE OUTCOME SCORES 
TO THEIR PROPER CELL (I.E.. ROW AND COLUMN INTERSECT). 
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3. OETEflMir€ WEIGHTING f^HOCEDURE OPTION TO BE USED WITH 
THEDCAM. 

A. GO WITH STATISTICALLY WEIGHTED NORMAUZED SCORES. 

B. HAVE A GROUP OF STAKEHOLDERS WEIGHT INFORMATION 
CATEQCM^IES or PERFORMANCE MEASURES QB 

C. STATISTICALLY REWEIQHJUOQMENTALLY WEIGHTED 
INFORMATION COMPONENTS OR PERFORMANCE MEASURES. 



4. RECALCULATE WEIGHTED PERFORMANCE MEASURE OUTCOME 
SCORES IF OPTION 3. B OR C IS ELECTED. 

5, RANK COURSES. 

A. TOTAL THE RAW OR WEIGHTED (I.E. RECALCULATED) 
PERFORMANCE MEASURE OUTCOME SCORES IN THE MATRIX 
ROWS LABELED WITH THE NAMES OR OTHER IDENTIFIERS 
FOR THE COURSES BEING ASSESSED. 

B. ARRANGE TOTAL SCORES IN DESCENDING ORDER COURSES 
OBTAINING THE HIGHEST TOTAL SCORES ARE CONSIDERED 
AS PERFORMING BEST. COURSES WITH THE LOWEST 
SCORES ARE CONSIDERED AS PERFORMING LEAST WEU AND 
THUS WOULD BE CANDIDATES FOR FURTHER REVIEW AND 
IMPROVEMENT INITIATIVES. 



6. REPORT ASSESSMENT FINDINGS. 



THE RECYCUNQ PROCBSS 

1 . REEXAMINE THE DCAM COMPONENTS AND REVISE THEM AS 
DEEMED NECESSARY OR APPROPRIATE. 

2. SELECT VOCATIONAL COURSES TO BE ASSESSED. AND CONDUCT 
THE DCAM. 
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The tasks rtquired to Install, operate, and recycle the DCAM can aslo be depicted In the following way: 



ir^ORMATtON 



SUBSYSTEMS 
SCORING 



RANKING 



Um 

INFORMATION 
COMPONENTS 

Fornmlata/RaYiw 

il^ORMATION 

CATEGORIES 

PERFORMANCE 
INDICATC^S 

PERFC^MANCE 
MEASURES 



F(Mtnulat«/R«via 
NORMAUZINQ 
SCHEME 

SETS OF 
I^MALIZED 
PERFORMANCE 
SCORES 



WEIGHT/VALUE 
PERFORMANCE 
MEASURES 

COMPUTE TOTAL 

SCORES FOR 

COURSES USING 

NON-WEIGHTED/ 

WEIGHTED 

»K3RMAUZED 

SCORES 

RANK COURSES 
(All Courses) 
(Within Clusters) 
(^twem Clusters) 



ACTIONS 



RANKINGS 
REVIEWED 

FURTHS? 
ANALYSIS 
CONDUCTED 

AOMIN 
ACTIONS 
TAKEN 

DCAM 
REVIEWED/ 
REVISED 
(IF NEEDED) 

NEXT 

ASSESSMENT 
SCHEDULED 
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CHAPTER III 



THE PILOT TEST OF THE DCAM 



Developme ntal Activities 

SELECTION OF COURSES FOR 
THE PILOT-TEST 

The Division of vocational and Career Education was 
responsible for selecting courses to be included in the 
pilot-test of the DCfM. Forty-one courses were selected 
by the division. 

THE INFORMATION SELECTION FRAMEWORK 

Project staff fonculated information categories for 
each of the framework's five information components. 
Then, performance indicators, measures and outcomes for 
each performance category were formulated. Project staff 
were aided in these tasks by examining the contents of the 
district's Reguest for Proposal which described many kinds 
of quantitative data about students and vocational courses 
that exist within the school district. 

The results were sent to the school district's 
Division of Vocational and Career Education for review and 
comments. Project staff then met in Cleveland with 
division staff to obtain their reactions. Division staff 
suggestions were incorporated into a revised set of 
performance categories, indicators, measures, and 
outcomes . 

The performance indicators in the revised set were 
sent out for review and comments to school district 
personnel employed outside the division. These persons 
were asked to suggest which performance indicators should 
remain or be deleted. They were also asked to suggest 
additional performance indicators that might be considered 
for use in the pilot-test. The responses were reviewed by 
project and division staff. The outcomes of the review 
were helpful in further revising performance categories, 
indicators, measures, and outcomes to be used with the 
DCAM adaptation being proposed to the division. 

Several more iterations in the development of 
information categories and performance indicators, 
measures, and outcomes were carried out by project staff 
before the pilot-test of the DCAM. The information 
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components and categories and the performance indicators, 
measures, and outcomes used in the pilot-test are found in 
appendix A. 

THE SCORING PROCESS 

Selecting A Mormaj-izing Procedure 

Project and division staff agreed that it would be 
practical to use a five-point scale to normalize 
performance measure outcomes. A Score of four would 
indicate the most desirable level of performance. A 
scores of zero would indicate the absence of a performance 
measure outcome. 

Weighting Information Com ponents 
And Performance Measures 

Judgmental and statistical procedures were used to 
assign weights of importance to information components 
and/or performance measures. Judgmental procedures used 
to weight information components and performance measures 
are described below, statistical procedures were used to 
determine the most efficient and powerful combination of 
performance measures. The use of statistical procedures 
is described in the section labeled Pilot-Test Outcomes. 

Judgmental Procedures 

The division convened a group of employers, school 
administrators, and vocational instructors. The convening 
served two purposes. It served as a forum for orienting 
key persons about the DCAM. It also enabled division and 
project staff to pilot-test a procedure for obtaining 
information component and performance measure weights. 

Project staff oriented the stakeholder group to the 
contracted scope of work with regard to the DCAM. Then 
the DCAM process was described. Lastly, the stakeholders' 
group was given instructions fc* weighting DCAM 
information components and performance measures. 

The stakeholders were told that the weights to be 
elicited from then would be obtained by a process of group 
consensus. They were asked to rank the information 
components so that the five components totaled 100 points. 
The participants recorded their judgements on forms 
provided to them. Then, each stakeholder informed the 
group about the weight they assigned each component. The 
weights were recorded and displayed. It was evident that 
their was little consensus among the stakeholders. 
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Project staff selected persons who assigned the 
highest or lowest weights to an information component. 
These persons were asked to explain the reasons for the 
weights they assigned to the components, stakeholders 
were encouraged to discuss the reasons why they felt 
particular information components were more important than 
others. They were then permitted to change the weights 
they initially assigned to each information component. 

After re-recording everyone's weights for the 
information components, the group agreed to accept the 
median of the group's weights for each information 
component for use in the pilot-test. The median weights 
for the information components is found in exhibit 13. 

The first set of performance measure weights did not 
require stakeholder involvement. Here, the number of 
performance measures within each information component was 
divided by the median weight assigned to the components by 
the stakeholders. The results of this weighting scheme 
are found in exhibit 14* In the exhibits that follow, 
performance measures are labeled as A through T. The 
performance measure statements A through T are found in 
Appendix A. 

Next, stakeholders were asked to weigh performance 
measures within each information component. In this 
instance, the sum of the weights they assigned to the 
performance measures had to equal the weight assigned to 
the information component of which the measures were a 
part. The same process of group consensus that was used 
to weight information com{x>nents was used to weight 
performance measures. The median weights obtained from 
the stakeholder group for thn performance measures are 
found in exhibit 15. 

Project staff believe that either of these two ways to 
get performance measure v^eights are acceptable. However, 
it can be argued that the latter procedure may more 
accurately reflect what stakeholders actually do during 
course assessment — even if they do so implicitly, if this 
assumption is correct, the procedure whereby stakeholders 
weigh performance measures within each information 
component may be the "best" or most sensitive one to use. 

Lastly, stakeholders were asked to weight the entire 
group of performance measures so that the sum of the 
weights totaled 100 points. Here they were looking at 
performance measures independent of information 
components. This was the most difficult weighting task 
because of the number of performance measures to be 
weighted (i.e. twenty ) . 
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Exhibit 1 3~lnf ormation Component Weights 



Cmtoxt/mput Prooessas 

Weight % [_ Jo) lZM- 

PrfmMaTm B C 0 E ' f" Q 
WtScora 



Exhibit 1 4 - Information Component And 

And Performance IMeasures Weights (Task 1 } 



Contaxt/taput Processes 

Weight % mm ^ 

PrfmMsfe ABCDE FQH 

Wt Score 4.0 4.0 4.0 4.0 4.0 2-8 2.8 2.8 



Exhibit 15- information Component And 

And Performance Measure Weights (Taslc 2) 



Cwitext/tnput Processes 

PrfmMsre * A 8 C D E FQH 
Wt Score 2.7 2.5 5.4 3.9 5.5 2.5 2.3 30 
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Outputs 



Tol 



OutcomM 

■ Q R 



Itenft 



i 



151 



Total 



OutCMitS 

/O 

'mm. 

o 

5.0 



P 

5.0 



3o /S' 

d R S T 
10.0 10.0 10.0 15.0 



Total 



Outputs 

/O 

6 p 

4.3 5.7 



OutcomM 

6 R 
10.2 12.9 



S 

8.9 



B«ift 

T 
15.0 



Total 
/CO 
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Again, the same process of group consensus that was 
used to weight information components was used to weight 
performance measures. Project staff felt that ^ , ^ 
insufficient time was available for this weighting task to 
he carried out satisfactorily. Therefore, this procedure 
for weighting performance measures was not included as 
part of the pilot-test. 



pilot-Test Outcomes 

COURSE RANKS 

Performance measure outcomes data for use with the 
DCAM were collected by division staff and sent to project 
staff for review and analyses. Project staff entered 
normalized performance measure outcome scores for the 41 
vocational courses included in the pilot-test into the 
appropriate cells of three ranking matrixes (see exhibits 
16, 17, and 18). 

Exhibit 16 depicts a completed ranking matrix that 
contains non-weighted normalized performance measure 
outcome scores Exhibit 17 depicts a completed ranking 
matrix that contains the information component and 
performance measure weights listed in exhibit 14. Exhibit 
18 depicts a completed ranking matrix that contains the 
weighted information components and performance measures 
found in exhibit 15. 

The sums of the non-weighted normalized performance 
measure outcome scores (exhibit 16) and the weighted 
normalized performance measure outcome scores (exhibits 17 
and 18) for each of the 41 courses in the pilot-test were 
computed. Course ranks were obtained by placing the 
summed scores in descending order. 

The three sets of course ranks that were produced were 
sent to division staff for review and comment. Project 
staff were informed that the DCAM pilot-test outcomes were 
generally consistent with staff judgements about which 
courses were performing best and which ones were most in 
need of a "get well" plan. 

Exhibits 16, 17, and 18 do not contain course names, 
courses are identified by number only. It would be 
insensitive to list course names for several reasons. 
First, the DCAM is being pilot-tested only. Second, data 
collection was incomplete. It was recognized at the onset 
of the pilot-test that the division would not be able to 
collect all of the data called for in the pilot-test. 
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Some performance measure outcomes used in the pilot' 
test were included even though it was known that data 
collection would be incomplete* However, division 
administrators anticipate that these data can be fully 
collected during the next school year. Ranks achieved by 
courses included in the pilot-test may change when 
revisions in the data set are made and data collection is 
complete . 

A REVISED INFORMATION SET 

Following the pilot-test, a revised information set 
(i.e., information categories, performance indicators, 
measures and outcomes) was produced. The revised 
information set is found in Appendix B. Prior to 
implementing the DCAM, division staff should review the 
revised set to determine that performance measure outcome 
data can be collected in a timely and reliable way. 

USING STATISTICAL PROCEDURES WITH PILOT-TEST DATA 

The purposes of using statistical procedures with DCAH 
data were as follows: 

o To define the extent to which 

statistical techniques weighted the 
performance measures differently than 
judgements by a group of stakeholders 

o To determine the most efficient and 
powerful combination of performance 
measures for predicting course 
"quality" . 

The outcomes that comprise each performance measure 
range in value from 0 to 4. They are all interpretable as 
interval level or above measures making them usable in 
correlation and regression designs. 

Exhibit 19 displays a frequency distribution of 
performance measure outcome scores where the performance 
measures Are not weighted. Some performance measures 
exhibit a near normal distribution of outcome scores 
(e.g., E and F while others are highly skewed (e.g., C) . 
Other performance measures show a decided tendency toward 
bi-modality (e.g.. A, C, and T) or approach near consensus 
(e.g., L). 



17 



Exhibit 16- A DCAM Ranking Matrix Using Raw Scores Only 
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ExhIbU 17 - A OCAM Ranking Matrix 
(Stakeholders' Task 1 ) 
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12.0 0.0 8.0 12.0 8.3 8.3 5.8 8.3 2.8 83 8-3 56 2.8 



ERIC 



48 



51 



3.0 2.0 
15.0 10.0 



3.0 
1S.0 



1.0 
SO 



20 4.0 2.0 

20.0 400 200 

ZQ 10 2.0 

20.0 10 0 20.0 



0.0 
0.0| 

o.o! 
o.o| 



46.0 
201.2 

44.0 
167.4 



2.0 3.0 
10.0 150 



2.0 2.0 
20.0 20.0 



00 
0.0 



2.01 
oO.Ol 



49.0 
215.2 



ZO 2.0 

10.0 lao 



2.0 1.0 0.0 1.0- 39.0 

2SJ.0 10.0 0.0 ISO- 185.8 



3.r 30 

15.0 150 

2.0 2.0 

Id 10.0 

2.0 3.0 

10.0 15.0 

3.0 4.0 

15.0 20.0 



2.0 4.0 0.0 
20.0 40.0 0 0 



2.0 2.0 
20.0 20.0 



2.0 2.0 
20.0 20.0 



0.0 
0.0 



2.0 4.0 0.0 
20.0 40 0 0.0 



00 

0.0 



0.0 
0.0f 

i 

o.ol 

o.o! 

i 

3.oi 

45.01 

i 
t 

o.ol 

0.0! 



42.0 
189.2 

43.0 
168.2 

51.0 
2499 

500 
200.4 



4.0 4.0 

20.0 20.0 

2.0 2.0 

10.0 10.0 



2.0 3.0 

20 0 30.0 

0.0 1.0 

0.0 10.0 



0.0 
0.0 

00 
0.0 



3.0! 
4S.0i 



! 

0.01 
OOi 



55.0 
261.7 

380 
129.6 



1.0 3.0 
5.0 15.0 



0.0 40 0.0 3.0j 41.0 

0.0 40.0 00 45.0I 200.6 



4.0 4.0 

20,0 20.0 

2.0 2.0 

10.0 10.0 



0.0 4.0 0.0 

0.0 40.0 00 

0.0 ZQ 10 

0.0 20.0 10.0 



30| 
45.0i 

i 

4,0| 
8O.0I 



51 0 

242.1 

39.0 
192.8 



2.0 2.0 0 0 2.0 2.0 2.0l 

10.0 10 0 0.0 20.0 20 0 30.0! 



32.0 

147.7 



2.0 4.0 
10.0 200 



0.0 2.0 0.0 I.Oj 
0.0 20.0 00 1501 



39.0 
160.6 



30 3.0 
150 15.0 



00 4.0 0.0 OOj 
0.0 40.0 00 O.OI 



35.0 
148.0 



3.0 2.0 
15.0 10.0 



0.0 4 0 
0.0 40.0 



0.0 
00 



2.0j 
30.0i 



430 

197.3 



49 5 . ) 



3lN(ymSc(mi 30 1.0 0.0 2.0 1.0 



NSxWS 



12.0 4.0 0.0 8.0 4.0 



30 
8.3 



30 
8.3 



ZO 
5.8 



2.0 
5.6 



0.0 0.0 
0.0 0.0 



30 
8.3 



2.0 
5.6 



10 
2.8 



aSNormScDral 30 30 0.0 4.0 2.0 2.C 2.0 20 2.0 0.0 30 30 2.0 1.0 

NSxWS i 12.0 liO 0.0 18.0 8.0 5.6 56 56 56 0.0 8.3 8.3 5,6 28 



33NonnScor«| 2.0 3.0 0.0 3.0 30 
NSxWS I 8.0 120 0.0 12.0 12.0 



34 Norm Scor« | 3.0 3.0 
NSxWS j 12.0 12.0 

35 Horn Swa 1 4.0 0.0 
NSxV'sJ I laO 00 

36 l>itorm Scora j 3 0 2 0 
NSxWS I 12.0 8.0 

i 

37 NDrm Score j 2.0 3 0 



NSxWS 



8.0 i; 



0.0 1.0 3.0 

0.0 4.0 12.0 

0.0 4.0 2.0 

0.0 16.C 8.0 

0.0 4.0 3.0 

0.0 16.0 12.0 

0.0 2.0 3.0 

0.0 80 120 



3.0 
83 

3.0 
8.3 

3.0 
8.3 

3.0 
8.3 

2.0 
56 



3.0 
8.3 

3.0 
8.3 

3.0 
8.3 

3.0 
8.3 

3.0 
8.3 



5.6 

2.0 
5.8 

2.0 
5.6 

2.0 
56 

2.0 
5.6 



20 
5.8 

2.0 
5.6 

30 
8.3 

2.0 
56 

30 
8.3 



0.0 
0.0 

2.0 
5.6 

20 
56 

0.0 
0.0 

2.0 
5.8 



3.0 
8.3 

3.0 
8.3 

0.0 
00 

3.0 
8.3 

3.0 
83 



3.0 
83 

30 
83 

3.0 
8.3 

3.0 
8.3 

30 
8.3 



20 
56 

2.0 
56 

2.0 
56 

1.0 
2.8 

2.0 
5.6 



1.0 
2.8 

1.0 
2.8 

1.0 
2.8 

1.0 
28 

3.0 
8.3 



38 Norm Score 
NSxWS 

39 Norm Score 
NSxWS 

40 Norm Score 
NSxWS 

41 Norm Score 
NSxWS 



3.0 2.0 

12.0 8.0 

4.0 1.0 

16.0 4.0 



ZO 

ao 

3.0 
12,0 



20 
80 



0.0 4.0 4.0 
0.0 16.0 16.0 



0.0 

0.0 



4.0 
•vO 



0.0 4.0 

0.0 16.0 



2.0 
8.0 



20 0.0 4.0 1.0 
8.0 0.0 1&0 4.0 



1.0 
♦ 0 



2.0 
5.6 

3.0 
8.3 

1.0 
28 

20 
5.6 



2.0 
56 



*Thi« sum <Joe» not include the scwe for performance measure S. 
Use S only when conparing courses whose completers are 
etig Ole for certification and licensing. 



2.0 
5-6 



2.0 4,0 

5.6 11,1 

1.0 4.0 

2.8 11.1 

2.0 40 

56 11.1 



30 
83 

4.0 

n.1 



20 
56 



2.0 
56 

0.0 
0.0 



30 
8.3 

3.0 
83 



20 0.0 4 0 
5.6 0 0 11.1 



0.0 
00 



30 
83 



30 
8.3 

3.0 
83 

3.0 
83 

30 
8.3 



2.0 
5.6 

0.0 
0.0 

2.0 
5.6 

2.0 
56 



1.0 
2.8 

1.0 
2.8 

10 
2.8 

10 

2.8 



ERIC 



50 



4.0 


30 


00 


1.0 


0.0 


3.0 




34.0 


1 RAfM 

t 

i 


COURSE NO. 






150 


00 


10.0 


0.0 


45.0 




16Z4 




] 


















1 1 

1 ^ 


22 


281.7 i 


2.0 


1.0 


0.0 


4..0 


00 


0.0 




34.0 


20 


249.9 1 


10.0 


S.0 


0.0 


20.0 


0.0 


0.0 




^302 


1 3 

\ 4 


25 
10 


24^1 1 
232.8 1 


3.0 


2.0 


o.o 


2.0 


0.0 


0.0 




37,0 


1 s 


8 


229.0 1 


15.0 


10.0 


0.0 


20.0 


0.0 


0.0 




141.S 


! ; 


9 


2186 1 


















18 


215.2 1 


3.0 


1.0 


0.0 


4.0 


0.0 


0.0 




39.0 


i 8 


37 


208.9 1 


150 


50 


00 


400 


00 


00 




158.3 


1 9 

i 

! 10 

I 


13 
3 


208.4 j 
205,9 i 


2.0 


1.0 


0.0 


1.0 


0.0 


1.0 




34.0 


I 11 


14 


201.2 i 


♦ W'V 


50 


00 


10.0 


0.0 


15,0 




1328 


¥ 13 


24 
21 


200,6 i 
200.4 i 


20 


2.0 


00 


3.0 


0.0 


3.0 




40.0 


1 « 


39 


199-6 i 


100 


10.0 


0.0 


30.0 


0.0 


45.0 


i 


1930 


,5 


5 


199.0 1 


















16 

i 


30 


1973 i 


20 


4.0 


0.0 


3.0 


0.0 


30 


t 

; 


450 


36 


193,0 i 


100 


200 


0.0 


30.0 


0.0 


45-01 


208.9 


i 18 


26 


192 8 i 














I 




i 19 


38 


192 6 i 


4.0 


2.0 


0.0 


1.0 


0.0 


3.0l 


43.0 


i 20 


IB 


1892 1 


200 


100 


0.0 


10.0 


00 


45.05 


192 6 


\ 21 


7 


188.2 i 














\ 




i 22 
1 23 


11 


187-2 1 




2-0 


00 


1.0 


00 


4.0i 


420 


6 


1856 1 




10^0 


0.0 


10.0 


0.0 


60.0; 


1996 


j 24 


1 


175 9 j 














i 
1 




! 25 


2 


172 3 j 


1 0 


20 


00 


3.0 


0.0 


ooi 


33.0 


! 26 


12 


172,0 1 




10 0 
t w«w 


00 


300 


00 


O.Oi 


1310 


i 27 


19 


168 2 \ 














j 

i 
t 




28 


15 


187 4 i 




^ 0 


0 0 


30 


0.0 


3.0 


390 


29 

i 


17 


185.8 1 




1$ 0 


0 0 


00 


00 


45.01 


157.8 


30 


31 


162.4 i 


















31 


28 


160 5 1 


















i 32 


34 


158.3 1 


















33 


41 


157,8 1 


















\ 34 


29 


1480 i 


















i 35 


27 


147 7 i 


















; 36 


4 


1443 i 


















i 37 


33 


141.8 1 


















i 38 


35 


132.8 j 


















1 39 


40 


1310 \ 


















j 40 


32 


130 2 j 


















j 41 


23 


129.6 j 



57 

51 



J 



Exhibit 18 ~ A I^M Ranking Matrix 
(Stakeholders' Task 2) 



CxttMt/tnput 



WaSght% 
Prfm Mw 
WtSoor* 



ABCDE FQH 

2.7 Z5 5.4 3.9 5.5 2.5 23 3.0 



1.9 



J 

^6 



K 
4.0 



U 
4.1 



M 
2.1 



N 
2.4 



Prfm MSfS 



B 



H 



M 



N 



CRS# 



1 Norm Scort 
NSxWS 



3.0 



m — »■ ■>■■ mn i ltt m»T - "f— ^-^ — I I •.■.•a — A^ 

4.0 0.0 3.0 3.0 4.0 4.0 



8.1 10.0 0.0 11.7 16.5 



10.0 9.2 



3.0 
9.0 



2.0 
3.8 



1,0 3.0 
2.6 12.0 



1 0 
4.1 



1.0 
2.1 



4.0 
9.6 



Norm Score i 
NSxWS 1 



3.0 
8.1 

4.0 
10.8 



3.0 
7.5 

3.0 
7.5 



0.0 4.0 

00 15.8 

00 4.0 

00 1S.6 



10 1.0 1.0 4.0 4.0 20 3.0 1.0 10 4.0 

5.5 2.5 2.3 120 7.8 5.2 12.0 4.1 2.1 1.8 

1.0 2.0 2.0 4.0 3.0 3.0 4.0 1.0 0.0 4.0 

5.5 5.0 4.0 12.0 5.7 7.8 18.0 4.1 0.0 96 



4 NormScor* 
NSxWS 



3.0 
8.1 



3.0 

7.5 



ao 
ao 



1.0 2.0 
3.9 11.0 



5 Norm Scots i 2.0 4-0 4.0 1.0 0.0 
NSxWS I 5.4 10.0 21.6 3.9 0.0 



1.0 1.0 4.0 3.0 

2.5 2.3 1Z0 5.7 

4.0 4.0 4.0 3.0 

10.0 9.2 1Z0 5.7 



3.0 4.0 1.0 
7.8 16.0 4.1 



1.0 4 0 
^8 16.0 



2.0 
82 



0.0 
00 

1.0 
21 



4.0 
9.6 

4.0 
9.6 



6 htormScorei 30 4.0 0.0 4 0 4.0 10 1.0 2.0 4.0 
NSxWS i 8.1 10.0 0 0 15 8 22.0 2 5 2.3 6.0 7.6 



1.0 30 30 4 0 1.0 
28 12 0 123 8.4 2.4 



Non ■ Scor* | 
NSxWS 



1.0 ^o o.e 2.0 4.0 3.0 3.0 20 3.0 

2 7 5.0 0.U 7 8 22.0 7.5 8.9 8.0 5.7 



1.0 3.0 30 4.0 4.0 
2.8 12.0 123 8.4 9-6 



8 Norm Score I 3 0 
NSxWS I ai 



9 NormSotve ; 
NSxWS { 

10 Norm Score I 



NSxWS 



3.0 

ai 

^0 
5.4 



11 NormSo(K9| 3.0 



NSxWS 



8.1 



2.0 0.0 30 3.0 4.0 4.0 1.0 3.0 

50 ao 11.7 16.5 10,0 92 3.0 57 

2.0 00 3.0 4.0 2.0 2.0 20 4.0 

5.0 0.0 11.7 22.0 5.0 4.8 6 0 7.6 

3.0 0.0 2.0 30 4.0 4.0 4.0 4.0 

7.5 0.0 7.8 18.5 100 92 liO 7.6 

3.0 0.0 20 2.0 30 30 2.0 4.0 

7.5 0.0 7.8 11.0 7.5 S.9 60 7.6 



1.0 
2.6 



3.0 
7.8 



0.0 3.0 
0.0 12.3 



1.0 0.0 3.0 
28 0.0 12.3 



0.0 3.0 
0.0 123 



3.0 0.0 30 
7.8 0.0 12.3 



1.0 
2.1 

40 

8.4 

4.0 
8.4 

4.0 
&4 



1.0 
2.4 

2.0 
4.8 

2.0 
4.8 

4.0 
96 



12 Norm Soore j 2.0 
NSxWS I 5.4 

13 Nwin Soor» 1 2.0 
NSxWS 1 5.4 



1.0 20 1.0 2.0 4.0 4,0 2.0 4.0 

2.5 108 31' 110 10.0 9.2 6.0 7.6 

2.0 2.0 3,0 2.0 3.0 3.0 2.0 4.0 

5.0 10.8 11,7 11.0 7,5 6.9 6.0 78 



20 3.0 3.0 2,0 3.0 

5,2 12,0 12.3 4.2 7.2 

20 3.0 3.0 2.0 30 

5.2 12.0 12.3 4.2 72 



ERIC 



52 



• * • • •Zsl 

> * • • 



0 
4.3 



P 

5.7 



UUTOOillvS 

Q n 

10.2 1^9 



s* 

6.9 



Bmft 

mm 

T 

15.0 



Total 




TOTAL 



3.0 ZO 

12.9 11.4 

2.0 4.0 

8.6 2Z8 



1.0 ZO 

4.3 11.4 

1.0 1.0 

4.3 S.7 

4.0 4.0 

17.2 22.8 

3.0 4.0 

12,9 228 



2.0 
8.8 



2.0 



1.0 

5.7 



4.0 3.0 

17.2 17.1 

3.0 2.0 

12.9 11.4 

3.0 3J0 

1Z9 17.1 

3.0 3.0 

12.9 17.1 



!.0 
5.7 



2.0 3.0 
8.6 17.1 



0.0 2.0 0.0 I.Oi 44.0 

aO 25.8 0.0 1S.0| 1738 

i 

I 

0.0 4.0 0.0 0.0| 42.0 

0.0 51.8 0.0 OO'l 177.1 



0.0 ZO 0.0 4.0| 44.0 

0.0 25.8 0.0 eO.oi 205.7 

0.0 4.0 0.0 OO'l 38-0 

0.0 51.6 0.0 O.oi 152.1 

i 

0.0 1.0 0.0 2.0! 

0.0 12.9 0.0 30.0| 199.2 

0.0 2.0 0.0 lOl 45.0 

0.0 25.8 0.0 15.0; 188.3 

0.0 2.0 0,0 3.0j 43.0 

0.0 25.8 0.0 45.0i 193.6 

0.0 4.0 0.0 4,0j 44.0 

0.0 51.8 0.0 BO.oj 234.5 

0.0 3.0 0.0 4.0i 44.0 

0 0 38.7 0.0 eO.oi 221.1 

0.0 4 0 0.0 3 0l 51.0 

0.0 51.8 0.0 45.0I 235.9 

I 

0.0 3.0 0.0 l.oj 46.0 

aO 38.7 0,0 15.3| 184.2 

I 

2.0 30 0.0 OOj 43.0 

20.4 38.7 0.0 0.0! 180.7 

2.0 2.0 0.0 2.0i 47.0 

20.4 25.8 0 0 3O.0I 214.7 



ERIC 



53 



5:i 



14 Norm Scon* ; 3.0 
NSxWS 8.1 



1.0 ZO 2.0 3.0 
2.5 10.8 7.8 16.5 



3.0 
7.5 



3.0 
6.9 



IS NwmSoaraj 2.0 3.0 20 



NSxWS 

16 Norm Scots 
NSxWS 



5.4 7.5 10.8 



2.0 3.0 
7.8 16.5 



2.0 2.0 2.0 4.0 2.0 
5.4 S.O 10.8 15.6 11.0 



17 NormSeor*] 3.0 3.0 2.0 2.0 2.0 



NSxWS I 

I 

18 NormScofit | 
NSxWS i 



19 Nc^Scm 1 



NSxWS 



\ 



20 riorm Scor« ! 



NSxWS 



21 Norm Score | 



NSxWS 



22 NormScora i 



NSxWS 



i 
I 
s 
: 
I 

23 ^tonlSco^» | 
NSxWS ! 

: 
t 

\ 

24 f^torni Score | 
NSxWS j 

5 
t 

26 Norm Score ! 
NSxWS I 

26 Norm Scots j 
NSxWS 1 



27 



i 

NormScofs I 
NSxWS 



i 



28 Norm Scots 
NSxWS 



I 



29 N(»m Scors 
NSxWS 

30 Norm Scots 
NSxWS 



I 



8.1 

2.0 
5.4 

1.0 
2.7 

3.0 
8.1 

2.0 
5.4 

3.0 
8.1 

4.0 
10.8 

4.0 
108 

3.0 
8.1 

2.0 
5.4 

3.0 
8.1 

2.0 
54 

10 
2.7 

3.0 
8.1 



7.5 10.8 7.8 11.0 



3.0 2.0 

7.5 108 

2.0 2.0 

5.0 10.8 



2.0 4.0 

7.8 22.0 

2.0 2.0 

7.8 11.0 



3.0 2.0 4.0 20 

7.5 10.8 15.6 110 

3.0 2.0 3.0 4.0 

7.5 10.8 11.7 22.0 

3.0 2.0 3.0 4.0 

7.5 10.8 11.7 22.0 

3.0 0.0 2.0 2.0 

7,5 0.0 7.8 11.0 

3.0 0.0 2.0 10 

7.5 0.0 7.8 5.5 

30 0.0 4.0 4.0 

7.5 OO 15.6 22.0 

2.0 00 3.0 3.0 

5.0 0.0 11.7 16.5 

1.0 0.0 3.0 2.0 

2.5 00 11.7 11.0 

2.0 00 4.0 2.0 

5.0 0.0 15.6 110 

1,0 OO 2.0 3.0 

2.5 0.0 7.8 18.5 

3.0 0.0 2.0 3.0 

7.5 0.0 7 8 16.5 



3.0 3.0 
7.5 6.9 



3.0 
7.5 

2.0 
5.0 

1.0 
2.5 

30 

7.5 

2.0 
5.0 

3.0 
7.5 

3.0 
75 

1.0 
2.5 

2.0 
SO 

2.0 
5.0 

2.0 
5.0 

2.0 
5.0 

40 

lao 

3.0 
7.5 

3.0 
7.5 



2.0 
4.6 

2.0 
4.6 

1.0 
Z3 

3.0 
6.9 

3.0 
6.9 

3.0 
6.9 

3.0 
6.9 

1.0 
2.3 

2.0 
4.6 

3.0 
6.9 

2.0 
46 

2.0 
4.6 

4.0 
9.2 

3.0 
6.9 

3.0 
6.9 



2.0 40 4.0 3.0 30 

6.0 7.6 104 12.0 123 

2.0 4.0 4.0 3.0 30 

6.0 7,6 10.4 12.0 12.3 

2.0 4.0 4.0 30 30 

6.0 7.6 104 12.0 12.3 



2.0 
6.0 

2.0 
6.0 

2.0 
6.0 

20 
60 

2.0 
6.0 

2.0 
6.0 

2.0 
6.0 

2.0 
6.0 

20 

6.0 

20 
SO 

ZQ 
6.0 

2.0 
6.0 

2.0 
6.0 

2.0 
6.0 



4.0 0.0 4.0 3.0 

7.6 00 18.0 12 3 

4.0 1.0 3.0 3.0 

7.6 28 12.0 12.3 

4.0 3.0 3.0 3.0 

7.6 7.8 12.0 12.3 

4.0 20 30 3.0 

7,6 5.2 120 123 

4.0 3.0 30 30 

7.6 7.8 12.0 12.3 

4.0 2.0 30 3.0 

7.6 52 12 0 12.3 

0.0 4.0 30 3,0 

0,0 104 12.0 12.3 

30 2.0 3.0 3.0 

5.7 5.2 12.0 12.3 

3.0 2.0 3.0 3.0 

5.7 5.2 12.0 12.3 

1.0 ZO 3 0 3 0 

1.9 5.2 120 12.3 



0.0 
0.0 
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Exhibit 19 - Frequency Distribution of 

Performance Measure Outcome Scores 
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Soae notion of those performance measures which 
discriminate most between courses can be inferred because 
of the construction of the item response structure. Given 
no weighting criteria and the fact that course **(|uality" 
is determined by a simple additive scale, key determinants 
of course quality would be a function of those performance 
measures that tend to be regarded as most importan\:. The 
higher the value of the mean, therefore, the greater its 
contribution to "quality". 

The frequency data suggest that those performance 
measures with the highest means would be the "most 
important". Hence, since performance measures A, D, F, G, 
I, K, and R are the perfor?nance measures with means above 
2.5, one can conclude that these measures alone could be 
used to determine course quality and the balance of the 
performance measures could be disregarded. 

To do this, would, however, undermine much of the 
value of the DCAM and would violate some of the 
fundamental assumptions of the behavior of variables in 
combination with one another. First, the DCAM process is 
designed to incorporate performance measures collectively 
for their combined contribution to "quality". Secondly, a 
frequency distribution is hardly the most effective means 
of identifying the mutual effects of variables on a 
dependent variable — in this case, performance measures on 
total performance measure score (TS). 

At the request of the sponsor, multiple linear 
regression was used tc determine the most efficient and 
powerful combination of performance measures for 
predicting course "quality", performance measures were 
entered as independent variables into a stepwise design 
with total performance measure scores being used as the 
dependent variable (see exhibit 16). 

Multiple linear regression was applied three times in 
order to satisfy the purposes of using statistical 
procedures. Exhibit 20 summarizes the results of multiple 
regression analysis where performance measures are not 
weighted and performance measure outcome scores range from 
0 to 4 (see exhibit 20) . Insofar as the critical feature 
of the design is relational strength, the absolute values 
of correlates will be reported for clarity. 

In each instance, the data snow the sequence of 
performance measures entering into the regression 
equation, an assessment of their strength of contribution 
upon entry, and their strength in the final summary, in 
accordance with the rules of stepwise regression, those 
performance measures whose level of statistical 
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significance was less than pa. 05 were deleted from further 
consideration. Nine of the nineteen performance measures 
included in the analysis contributed significantly to the 
dependent variable. These are reported in exhibit 20. 

Multiple Regression Where Performance 
Measures Are Mot Weighted 

The output from using multiple linear regression 
reveals that not all performance measures are of equal 
value in predicting course quality. Only about half of 
the performance measures were needed to "explain" 93.4 
percent of the variance in t^e dependent variable. Thus, 
it can be fairly stated that: the application of the DCAM 
using non-weighted performance measure outcome scores did 
a good job in accounting for differences in quality among 
courses . 

Another piece of information provided by the data 
concerns the relative value of variables within the 
equation. It will be notud that the sequence of variables 
entering into the regression equation were a function of 
their relative strength as parti^.s. In other words, the 
magnitude of their coefficients uutside of the regression 
equation determined their entry. Yet, one advantage of a 
regression design is that this strength is modified upon 
the entry of other variables, either strengthening it in 
the case of an interaction effect or weakening it as a 
function of the latter factor explaining the variance 
better. Therefore, exhibit 20 shows some differences in 
the Beta values in terms of the entry of other variables. 
We discover, then, that performance measures G, T, and I 
are ultimately the most important performance measures 
with regard to predicting total score (i.e., course 
"quality"). 

These findings have several meanings. It could mean 
that course cost per completer, entry-level wages paid to 
course completers obtaining training related jobs, and 
gender and racial balance on course advisory committees 
are the most important individual performance measures in 
determining course quality. Alternatively, it may simply 
mean that process measures in general are the more 
important factors along with the wages that completers can 
command in training related jobs, certainly, some cogent 
explanation could be developed for why these particular 
performance measures emerge. Yet, no such explanation 
seems obvious. Thus, it could be that these measures 
emerged purely as an artifact of the conditions of 
analysis (i.e., that every performance measure is treated 
equally.) Because we have stakeholder data o this point, 
subsequent analyses may clarify the situation for us. 
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Multiple Begyessio ""- stakeholders Tagk I 



Exhibit 21 suwiaarizes the results of multiple 
regression analysis where information components were 
weighted by a group of stakeholders as described in a 
preceding section and performance measures are weighted as 
a fraction of the weight assigned to their component, 
(see exhibit 17). The recalculated performance measure 
outcome scores were then obtained by multiplying the 
performance measure weight by the raw outcome score 
associated with the measure. 

This regression analysis shows several similarities 
with the preceding one. First, the strength of the 
equation is also very high, even somewhat enhanced. This, 
in part, is an artifact of the composition of the 
dependent variable. However, it is useful to show that a 
subset of variables are the primary ones in explaining the 
variance in Total Score in contrast to the use of the 
entire list. Additionally, one of the crucial variables 
in the equation is the same— the entry level wages earned 
by course completers obtaining training-related 
employment . 

Whatever the similarities, the differences may be more 
important here. Performance measure T as an independent 
variable is much stronger in this equation. Further, one 
notes that performance ajeasure R, the extent to which 
course completers are working in training-related jobs, 
are in the military, or are pursuing further education, 
plays an important role here while not even making an 
entry in the first analysis (see exhibit 20). Indeed, the 
logic of variables T and R performing key roles in 
determining course quality makes a good deal better 
intuitive sense as differentiating measur&s thus sparing 
us from the somewhat stretched logic the previous analysis 
offered. Additionally, the stronger variables in this 
equation seem to complement one another conceptually in 
contrast to what we saw where performance measures were 
not weighted. 

Finally, considerably more variables enter the 
equation itself. This fact suggests that the differential 
weighting of their information components changed the 
analysis in other ways as well. By inspection, the 
somewhat lesser contribution of process variables and the 
greater role of context, output, and outcome variables 
suggests an overall improvement in the interpretability, 
hence, usability, of the model. This point can be explored 
in view of the thiri analysis. 
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Multiple Regression— Stakeholders Task 2 



Exhibit 22 summarizes the results of multiple 
regression analysis where performance measures were 
differentially weighted by the stakeholders and the sum of 
the differential weighting equalled the weight assigned to 
their information component (see exhibit 18). Again, 
recalculated performance measure outcome scores were then 
obtained by multiplying the performance measure weight by 
the raw outcome score associated with the measure. 

The data in exhibit 22 show some many of the same 
tendencies exhibited in the preceding analysis— albeit 
somewhat stronger. All five information components 
influence quality to some extent. Benefit, outcome, and 
output variables emerge earliest and generally more 
strongly. Benefit and outcome measures are the most 
influential in defining course quality and this finding is 
intuitively sensible, it can be argued that better course 
outcomes and benefits depend on effective educational 
pt*rtcesses and context conditions as the DCAM model 
prev^cts. This sustaining of the overall DCAM approach is 
prirhfps the most important finding that results from this 
Xl^^ii ct Statistical analysis. 

Lj.mitations o t Thg Finainqg and Conclusions 

One limitation in the application of the statistical 
procedures using pilot-test data is that these data were 
unavoidably incomplete. With this limitation in mind, two 
conclusions can be drawn from the statistical analyses 
presented above. First, the DCAM approach can be 
statistically validated as it has been in this review. 
Second, the DCAM generally works as it is proposed and it 
works optimally with stakeholder involvement. The data 
lend empirical support for the involvement of stakeholders 
as well as showing the relative contribution of 
information components and performance measures within the 
model . 
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Exhibit 20 -> Stepwise Multiple Regressaion 

(Non-Weighted Performance Measures) 
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Exhibit 21 Stepwise Multiple Regresssion 
(Stakeholders' Task 1) 
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Exhibit 22 - Stepwise Muitlple Regresssion 
(Stal(8holders* Task 2} 
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ASSESSING COURSE PERFORMANCE 
7HB INFORMATION FRAMEWORK 



INPUT/CONTEXT COMPONENT 



Information Category: Enrollment Equity 



Performance indicator 1 

The extent to which there was equity in enrollment of B\ack students In vocational education 
courses. 

Performance MMSure A and Scores: 

[4] This course had the best record In its career duster with regard to first-year 
enrollment of Blaci( students (i e.. closest to the goal of 70% Btacit students) 

[3] Between 55.0 and 64.9 percent or between 75.1 and 85.0 percent of the first- 
year students enrolled in this course were Black. 

{2} Less than 55.0 or more than 85 percent of the first-year 
students enrolled in this course were Black. 

1 1 ] This course had the greatest inequity In its career cluster with 
reg»'d to first-year enrollment of Black students (I.e., furthest 
from the goal of 70% Biad( students). 

[0] No or insufficient data are available for this course. 



Performance IndlcAicr 2 



The extent to which there was sex equity in enroilmen^a In vocational courses. 
Per for/nance Mewsurs B and Scores: 

[A] This course had the best record In Its career cluster with regard 
to sex equity In ervollment of first-year students (i.e.. closest to 
the goal of 50% for both sexes). 

I3J This course was above the median for courses In Its career 
cluster with regard to sex equity In enrollment of first-year 
students (i.e.. more than 1/2 of the courses were further from 
the goal of 50% for tioih sexes). 

(2] This (^urse was at or below the median for courses in Its career 
cluster with regard to sex equity In enrollment of first-year 
students (i.e., 1 /2 or more of the course were closer to the 
goal of 50% for both sexes). 

[ 1 j This course had the greatest inequity In Its career cluster with 
regard to sex equity for enrollment of first-year students (I.e., 
furthest from the goal of 50% for both sexes). 

[0] No or insufficient data are available for this course. 
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information Category: Acc&ss to Courses 



PerfCM'mance indicator 3: 

The extent to which students are able to enroll In a vocational course of their choice. 
PBrft^rmnce Measure C and Scores.' 

t4J This vocational education course was one of the three having 
the highest percent of students able to enroll in a course that 
was their first choice. 

13J Compared with other vocational courses, this course ranked 
above the median with regard to percent of students able to 
enroll in a course that was their first choice. 

12] Compared with other vocational courses, this course ranked at 
or below the median with regard to percent of students able to 
enroll in a course that was their first choice. 

[ 1 1 This vocational course was one of the three having the lowest 
{i^rcent of students able to enroll in a course that was their first 
choice. 

[0] No or insufficient data are available for this course. 
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Performance indicator 4: 

The extent of opening enrollment to first year course capacity. 
Perfwrnance Measure D and Scores: 



[4J Tills v(K»tionai education course was one of the three with the 
high^ p^cent of opening enrollment to first year course 
cap^ity. 

[3] Compaf ed with other vocational courses, this course ranked 
above the median with regard to percent of opening enrollment 
to first year course capacity. 

12] Compared with other vocational courses, this course ranked at 
or below the median with regard to percent of opening 
enrollment to first year course capacity. 

[ 1 ] This vocational course was one of the three with the lowest 
percent of opening enrollment to first year course capacity, 

[0] No or Insufficient data are available for this course. 



Performance Indicator 5: 

The extent of enrollees returning for a second year of instruction. 
Pertormmce Measure B and Scores: 

[4] This vocational education course ranked one of the three 

having the highest percent of first-year students returning to the 
course fix a ^cond y^ of Instruction. 

[3] Compared with other vocational courses, this course ranked 
above the median with reganj to percent of first-year students 
returning to the course for a second year of instruction. 

[2] Compared with other vocational courses, this course ranked at 
or below the median with regard to percent of first-year 
students returning to the courae for a second year of 
Instruction. 

{ 1 ] This vocational course ranked one of the three having the 
low^ percent of first-year students returning to the course for 
a second year of Instruction. 

[0] No or insufficient data are available for this course. 
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PROCESS COMPONFNT 



Information Cate{K>fy' Cours0 Costs 



Performance Indicator 6: 

Relative costs associated with operating vocational courses. 

Performance Measure F and Scores: 

[4] Compared with the other vocational courses in its career 
cluster, this course had the lowest per-student operating cost 
before reimbursement. 

[3] Compared with the other vocational courses in its career 
cluster, this course ranked below the median with regard to 
per-student operating cost before reimbursement. 

{2] Compared with the other vocational courses in its career 
cluster, this course ranked at or above the median with regard 
to per-student operating cost before reimbursement. 

[ 1 ] Compared with the other vocational courses in its career 
cluster .this vocational course had the highest per-student 
operating cost before reimbursem^ . 

[0] No or insufficient data are available for this course. 
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Performance MeasurB G arni &:ores: 



[A] Compared with the other vocational courses In Its career 
cluster, this course had the lowest cost per graduate. 

{31 Compared with the other vocational courses In Its career 
cluster, this course ranked t>elow the median with regard to 
cost per graduate. 

[21 Compared with the other vocational courses In Its career 
cluster, this course ranked at or alwve the median with regard 
to ct»t per graduate. 

( 1 1 Compared with the other wcatlonal courses In its career 
ciuster.thls course had the highest cost pt. graduate. 

[0] No or insufficient data are available for this course. 



Information Category: Private Sector Support 



Performance indicator 7: 

The extent to wWch private sector jources cwtrlbuted to the operations of vocational 
courses [Note: Only applies to courses in magnet schools]. 

Performance Measure H aiKi Scores: 

[4] This Is at least tho second corwecutive year that one or more 
outside agencies donated funding, equipment, supplies, 
instructional materials, or other forms of tangible support god 
at least one of these agencies participated in the course 's 
advise^ committee both ^is year and last year. 

[3] This Is the first year that one or more outside agencies donated 
funding, equlpmem, supplies. Instrutnior. I materials, or other 
forms of tangible supi^ aSMi at least one of these agencies 
participated in the (purse's advisory ccHnmittee this year. 

[2] Outside agencies contributed to this course this year only by 
participating in the course's advisory committee. 

1 1 ] Outside agencies did not contribute to this course this year 
either by contributing tangible forms of support or by 
participating In the course's advisory committee. 

[0] No or Insufficient data are available for this course. 
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Performance Indicator 8: 

The extent of female and minority participation on course advisory committees 
[f*5te: Only applies to courses In magnet schoolsl. 

P0rformanc9 Msasure I and Scotbs: 

[A\ The membership of the course advisory committee Included 
both sexes as well as more thar - minority member. 

{31 The membership of the course advisory comnlttee Included 
both sexes and one of these persons Is a Black. 

[21 The meml^rshlp of the course advisory committee did not 
Include both sexes but at least one member was Black. 

1 1 1 The membership of the course advisory committee were all 
menders of the same sex and non of them were Black. 

[01 There was no active advisory committee or no or Insufficient 
data are available for this course. 
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Information Category: Secondary-Postsecondary ArtlculaUon 
Performance Indicator 9: 

The extent of secondary-postsecondary articulation for course credit. 

PerfornwJCB Measure J and Scores: 

[4] A wrhien articulation agreement exists enabling course 
graduates to r«ceive advanced standing or course credit at a 
postsecondary institution. 

{3] Efforts were ongoing to obtain a written articulation a&reement 
with a postsecondary Institution that would enable course 
graduates to receive advam:ed striding & course cr^it 

[2] There was written evidence that school staff are planning to 
initiate efforts to obtain an anicuiatlon a^^eement with a 
|x>stse«>ndary institution that would enable course graduates 
to receive advanced standing or course credit. 

[ 1 1 There was no wi Itten evidence that school staff were planning 
to initiate efforts to obtain an articulation agreement with a 
postsecondary institution. 

[0] IMo or insufficient data are available for this course. 
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Information Category: Professional Development 



Performance Indicator 10: 

The extent of Instructors' jMrtlclpatlon In professlona! development experiences. 

P^formance Measure K ^ Scores: 

[4] At least one instructor for this vocational course participated in 
more than one formal professional development experience 
this year. 

(3] At least one Instructor for this vocational course participated in 
one formal profe»ional development experience this year. 

[2] None of ^e Instructed for this vocationai course panicipated 
in a formal profe^ionai development experience this year. 
Howe^, at least one instructor currently assigned to this 
vocatic^ course did so last year. 

[ 1 ] IMone of the instructors for this vc^ational course* participated 
In a forntal professional development experience this year or 
lasty^. 

[0] No or insufficient data are available for this course. 
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!nfct illation Category: Instructionai Design 



Performance Indicator: 11: 

The extent to which vocational course curricula are competency based. 

Pertormsnce MsasurB L and Scores: 

[4] The vocational course curriculum offered this year was 
considered by its division supervisor to iae fully competency 
based. 

|3] The vocational couree curriculum offered this year was 

cor^dered by its division supervisor to be partly competency 
b^ed. Activities are und^way to mai<e the curriculum fully 
competency based. 

[2] The vocational course curriculum of ered this year was 
consider«j by its division supervisor to be partly competency 
based. Activities were ogi underway to make the cun-icuium 
fully competency based. 

11} Th 3 wcationai curriculum offers this year was considered by 
its division supervisor as not being even partly cor'petency 
based. 

[0] No or insufficient c'ata ar*» available for this course. 
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Performance Indicator 12: 



The extent to which vocational courses include the use of computer software in the 
Instructional process. 

Psrtwmancc Measure M and Scores: 

[4] Hiis was at least the second consecutive year that students in 
this vocational course used computer software to teach 
empioymem and/or empio^'abiilty skills. 

[3] This was the first year that students in this vocational course 
used computer software to teach employment and/or 
employabiiity sidlis. 

[2] This vocational course did not include the use of computer 
software to teach employnient and/or impioyabiilty siciiis this 
school year: iKSwever, there Is written eviderK^e that such 
ir^ruction will oe included next »:hooi y>f ar. 

f 1 ! This vocational course did not include the l se computer 
software to t^ach employment and/or employabiiity sisiiis this 
sc^ x)l yesr b'"^ there 's n-j written evidence that there are 
•jians to do k> next school year. 

1 01 No or Insufficient da*a are available for th}« course. 
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Information Category: Student Organization Participation 
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Performance indicator 13: 

The extent of participation of students In vocational education student organizations. 
Performmice Measure N and Scores: 

[A] Compared with the other vocational courses In Its career 
cluster, this course had the highest percent of student 
participation in a vocational education student organization. 

(3] Compared with the other vocational courses in its career 
cluster, this course raniced above the m^ian with regard to the 
percent of student participation in a vocational education 
stud^t organization. 

[2] Compared with the other vocational courses in its career 
cluster, this course ranited at or below the median with regard 
to the percem of student participation in a \^>catlonai 
^ucation student organization. 

[ 1 1 Compared with the other vocational courses In its career 
cluster, this course had the lowest percent of its students 
participate in a vocational education student or^izatlon. 

[0] No or insufficient data are available for this course. 
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Oin PUT COMPONENT 
Information Category: Course Attrition /Completicm 
Performance indicator 14: 

Hie extent of student dropout from vocatic.ial education courses. 

Performance Measure O and Scores: 

[4] This vocational education or<jr»s was one of the three having 
the lowest percent of dropouts this year. 

[3| O^mpared with other vocational course, this course ranited 
ijelow the median with regard to percent of c-ropouts this year. 

(2] Compared with o^er vocational courses, this course ranited at 
or atove the median with re^rd to percent of dropouts this 
yeau-. 

{ 1 ] This vocational course was one of the three having the highest 
percem of dropouts this year. 

[Oi Ho or insufficient data are available for this course. 




Performance Indicator IS: 



The extent of graduates to course capacity 
Performance Measure P and Scores: 

I4J This vocational education course was one of the three having 
the highest percent of graduates to course capacity. 

[31 Compared with other vocational courses, this course ranked 
above the median with regard to percem of graduates lo 
course capacity. 

121 Compared with other vocational courses, this course ranked at 
or below the median with regard to percent of graduates to 
course capacity. 

[ 1 1 This vocational course was one of the three having the lowest 
percem of graduates to course capacity. 

[01 No or Insufficient data are available for this course. 
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OUTCOME COMPONENT 



Information Category: Job Placement 



Performance Indicator 16: 

The extent of training-related job placement assistance to students. 
Performance Measure Q and Scores: 

[4] This vocational education c.^^ se was one of the three with the 
highest percent of students placed in training-related jobs with 
help from school district staff. 

[3] This vocational course ranked above the median with regard to 
students placed In training-related jol» with help from school 
district staff. 

[2) This vocational course ranlced at or below the median with 
regard to students placed in training-related jobs with help 
from school district staff. 

{ 1 ] This vocational course was one of the three with the lowest 
percent of students placed in training-related jobs with help 
from sciiool district staff. 

[0] No or insufficient data are available for this course. 
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Perfwinance indicator 17: 



The extent to which course graduates are either working In training-related jobs, are 
military, or are pursuing further education. 

t'erfornmnce Measure R and Scores: 

[4] This vocational education course was one of the three with the 
highest percent of students currently In training-related jobs, in 
the military, or pursulpj further educatlwi. 

{3J This vocational course ranked above the median with regard to 
the percent of students currently In training-related jobs. In the 
military, or pursuing further education. 

12 J This vocational course ranked at or below the meolan with 
regard to the percent of students currently In training-related 
jobs. In the military, or pursuing further education. 

[ 1 ] This vocational course was one of the three with the lowest 
percent of students currently In training-related Jobs. In the 
military, or pursuing further education. 

[OJ No or Insufficient data are available for this course. 
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Information Category: Ucenalng/certlflcatlon 



Performance indicator 18: 

The extent of llcenslng/certlflcatlwi of course graduates [when applicable]. 
PerformancB Measure S and Scores; 

[41 Compared with the other vocatloi»l courses In Its career 
cluster, this vocational education course had the highest 
percent of its graduates obtain a formal license or professional 
certlfk^te. 

[3] Compared with the other vocational course in Its career 
cluster, this vocational education course ranked above the 
median with regard to the percent of graduate obtaining a 
formal license (h* professional certification. 

[2] Compared with the other vocational courses in its career 
cluster, this vocational education course ranked below the 
median with regard to the percent of ^aduates obtaining a 
formal license or profe^lonai certification. 

1 1 ] Compared with the other vocational courses In its career 
cluster, this vocational education course had the lowest 
percent of graduates obtain a formal license or professional 
certification, 

[01 No or Insufficient data are available for this course. 
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BENEFITS COMPONENT 



Information Category: Entry Wr.ges 



Performance Indicator 19: 

The extent of entry-level wages of course graduates obtaining training-related jobs. 
Performance Mea^e T and Scores: 

[4] Compared with the other r^-atlonal courses In Its career 
cluster, graduates from this vocational education course 
working in training^elatad Jobs obtained the highest {median} 
entry tev^ wa^ ^m their ^pioyers. 

{3] Compared with the other vocational courses in its career 
cluster, ^aduates from this vocational education course 
working in training-related jobs ranked above the median for 
[median] entry level wages from their employers. 

[2] Compared with the other vocational courses In its career 
cluster, graduates from this vocational education course 
working In trainingrelated jobs ranked below the median for 
Imedlan] entry level wages from their employens. 

( 1 } Compared with the other vocational courses in Its career 
cluster, graduates from this vocational education course 
working in training-related jobs obtained the low^ (median] 
entry level wages from their employers. 

[0] No or insufficient dau are available for this course. 
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APPENDIX B 



REVISED INFORMATION SET 
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THB INFORMATION FRAMEWORK 



INPUT/CONTEXT COMPONENT 
Information Category: EnrollmBnt Equity 



Performance Indicator 1 

The extent that there is equity in enrotlment of Black students in vocational education 
courses 

Performance Measure A 

The percent deviation from the district's goal that Black students represent seventy percent 
of each vocational course's first-year opening enrollment.^ 

PerformancB Measure Outcomes and Scores (where (41 Is best performance) 

[A] This course ranks anK>ng the three deviating least from the 
district's goal for Black student enrollment. 

{3] This course ranks above the m^iian for deviating least from the 
dlstrtet's goal for Black student enrollment^ 

[2] This course ranks at or below the median for deviating least 
from the district's goal Black student enrollment. 

1 1 J This course ranks anK>ng the three deviating most from the 
district's goal for Black student enrollment.^ 

[0] Performance data are not available. 

^The district's goa\ is the curr«n racial balai'ice of school 
enrotlm^t (i.e., seventy percent Black). Vocational courses 
^uld be ranked according to their deviation from seventy 
percent The seventy percent figure needs to be reviewed 
periodically and adjusted as necessary. 

^in this and similar statements the phrase 'above the median' 
does not include the top ranked courses in [4]. 

^in this and similar statements the phrase "below the median' 
does not include the bottem ranked curses In [1 } 
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Performance Indicator 2 



The extOTt of sex equity in ^rolimaits in vocational o^urs^ 
Performance Measure B 

The percent deviation from the goai that female students represent fifty percent of each 
vocational course's first-year opening enrollment.^ 

PmrfiAmsncB Measure Outcomes and Scw9S (when {4] Is bsst performance) 

[A] This course ranks first In Ite career clusttjr for least deviation 
from the goai for fen^le student enrollment. 

[3] This course ranks atxjve the median in its career cluster for 
least deviation from the goal for female student enrollment. 

{2] Th's course ranks at or below ^e median In its career cluster 
for least deviation from the goai for female student enrollment. 

{ 1 ] This course ranks last in its career cluster for least deviation 
from the ^al for female first-year opening enrollment. 

[0] Performance data are not available. 

Vocational courses should be ranked according to the district's 
goal for female enrollment in vocational courses (e.g.. fifty 
percent). The f iftv percent figure needs to be reviewed 
periodically and adjusted as necessary. 
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Information Category: Course Popularity 



Performaroe Indicator 3: 

The extent that students enroll in a vocational course of their choice 
Performance Measure C 

The percent of first-year students in a vocational course selecting it as their first choice 

PertormMce Measure OutCMnes and Scores (where [4J /s best performance) 

[4] This course is among the three with the greatest percent of 
first-year students selecting it as their first choice. 

(3] This course ranlts above the median for the percent of first-year 
students selecting it as their first choice, 

\2] This course ranks at or below the mp Jian for the percent of 
first-year studwts selecting it as thd." first choice, 

[ 1 ] This couree is among the three with the smallest percent of 
first-year (^..udents selecting it as their firs^ choice. 

[0] P«tormance data are not available. 




Performance indtcator 4: 



The extent that vocational courses make use of training capacity 
Performance Measure D 

The percentage tfiat results from dividing enrollment of first-year students by enrollment 
capacity. 

PBrfwmancB Measure Outcomes and Scores (wlmre {41 is best performance) 

[4] This course ranits among the three with the greatest percentsge 
resulting frwi dividing enroiiment of first-year students by 
^roilment capacity. 

{3} This course ranl<s above the median for the percentage 
resulting from dividing enroiiment of first-year students by 
enroiiment capacity. 

[2] This couri^ ranits at or t^iow the median for the percent^ige 
resulting frnm dividing enroiiment of first-year students uy 
enroiiment capacity. 

[ 1 1 This course ranks among the three with the smallest percentage 
resulting from dMding enrollment of first-year students by 
enrollment capacity. 

[ui Performance data are not available. 
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Pdrformance indicator 5: 

TTw extent that first-year vocatit^t students return to the same course for a second year of 
instruction 

Performance IMeasure E 

The percent of f Irst-^ar vocationai students who return to the s=me course for a second year 
of instruction 

Performanco Measure Outcomes and Scores (where / 4/ is t>est performance) 

[A] This course rani<8 arTK)ng the three with the greatest percent of 
first-year students who return to the same course for a second 
yearof instnK:ti(m. 

{3] This coutse ranlcs above the median for greatest percent of 
first-year students who return to the same course for a second 
year of instruction. 

(21 This course ranits at or below the median for greatest percent 
of first-year students who return to the same course for a 
second year of instfuctton. 

1 1 } This course ranlts among the three with the smallest percent of 
first-year students who return to the same course for a second 
year of Instruction, 

[0 j Performance data are not available. 
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PROCESS COMFONENT 



Information Cate^ry: Course Costs 

Performance Indicator 6: 

Costs associated with operating vocatiunal courses 

Performance Measure F 

The per-sfjdem operating costs for wcational courses^ 

P9rformmiCB hUeaaure Outcomoa and Scoros (wtwre {4J Is desr fwrformance) 

[4] This course has the lowest per-student operating cost in Its 
career cluster. 

{3] This course has a per-student operating cost that is at or below 
the median in its career cluster. 

(2) This course has a per-student operating cost that Is above the 
median in its career ciust^. 

{ 1 ] This course has the higliest per-student operating cost in its 
career cluster. 

[0] Performance data are not available. 

^The calculation of per-student operating cost does not Include 
state reimbursement. 
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Performance Measure 6 

The per-completer operating costs for vocational courses^ 

Pertormmtee MoamitB Outcomss and Scores (whon [4J is best performance) 

(4] This &3wse has the lowest per-compieter operating cost in its 
carei^ cluster.. 

{31 TWs course ranks above the median In Its career cluster for per- 
conipleter operating costs. 

12J This course ranks at or t>etow the median In its career cluster 
for per-^ompieter jperattng cost. 

{ 1 J This {^rse hh. t ie highest per-completer operating cost In its 
career clust'rf. 

[0] Perfofntance data are not available. 

^The calculation of operating cost per completer does not 
include state reimbursement. 
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Inforr lailon Category: Private Sector Support 



Performance Indicator 7: 

The extent that pfh«te sector sources contribute to the operations of vocational courses 
Performance Measure H 

The presence of tangible support (e.g.. funding, equipment, personnel) by private sector 
sources 

Performance Measure Outcomes and Scores {where {4} Is best performance) 

[41 More than one private sector source gave tangible support to 
this cour^ this y^r and last year. 

[3] Only one private sector source gave tangible support to this 
course this year and last year. 

(2] Only one private sector source gave tangible support to this 
course this year. Their was no tangible support given by any 
private sector source last year. 

[1 1 No private sector source gave tangible support to this course 
this year or last year. 

[0] Performance data are rK>t available. 
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Performance indicator 8: 

lh9 extern of female and minority participatK^i on vocational covirse advisory committees 
Performance Measure I 

The nun^ of pemm$ from each sex and tne number of minority persons represented on 
vocationai course advisory committees 

PBfformance Measure OutcomBS and Scores f w/iere [4 J is best performance) 

{4] The memb«^ip of the course advisc^y committee includes 
both males and femalm. Two or more members are minority 
persor^ 

{3} The membership of the course advise^ committee includes 
both males and females. Only one member is a minority 
person. 

[2] The memt»rship of the course advisory committee includes 
only males ar only female. Two or more members are minority 
persOTS. 

{ 1 ] The membenship of the course advisory committee includes 
only males or only females. Only one of them is a minority 
person. 

[0] The mmnbership of the course advisory committee include 
only males or only females. None of them is a minority person. 
The committee is not active or performance data are not 
available. 
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inforinatiofi Category: Pc^tsecondaty Credit 



Performance Indicator 9: 

The presence of an articulation ageement with a postsecondary institution enabling 
studeni^ to obtain credit for completed course worl< 

Performance Measure J 

The extent to which an articulation agr^ment is in force ^ is in the worths 

Pmrtomanc9 Measure Outcom9$ and Scores (where [4] Is best performance) 

[4] An articulation a^eement is in force for counse completers to 
receive (Ksstsecondary institution credit for course worl< 
already completed. 

(3] An articulation agreement will be in force next school year for 
course completers to receive postsecondary Institution credit 
for course work already ccHnpieted. 

[2] An articulation agreement is in the worics but will not be in force 
during the next school year. When it is in force, course 
completers will be able to get postsecondary Institution credit 
for couree work already completed. 

[ 1 ] There are no efforts in the works for an articulation agreement 
for completer to receh/e postsecondary institution credit for 
course work already completed. 

[0] Performance data are not available. 
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information Category^ Profes^onal envelopment 



Performance Indicator 10: 

The extmt that iretructors participate in professional development activities 
Performance Measure K 

Whether vocatlr^il instructors participated in professional development activities this year 
and last year 

Performance Measure Outcomes and Scores (where [4] Is best performance) 

[A] At least one instructor for this course participated in more than 
one formal professional development activity this year. 

{3] At irast one instructor for this course particif^ted in a formal 
professional development activity this year. None of the 
instructons for this course participated In more than one. 

{2] None of the Instructors for this course participated in any 
ft)rmai profftssionai development activities this year. At least 
one of them did so last year, 

1 1 ] None of the instructors for this course participated in a formed 
professional development activity this year or last year. 

{0] Performance data are not available. 
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Information Category: Instructional Design 



Performance Indicator 1 1: 

The extent that the vocational course currlculuni is competency based according to state 
department of education standards 

Performance Measure L 

Competency-k)ased vocational curriculum is In force or is in the worics 

Performance Measure Outcomes and Scores (where (4] Is best performance) 

{4} 'mis couree has a competency-tMsedcun'iculum that meets 
state department of education standards. 

[3] The ojurse cufflculum is partly competency based according to 
its division supervisor. Curriculum development activities are 
currently in progress to meet state competem:y't»sed 
st»idards. 

121 The course cun-lcuium Is partly competency based according to 
its division supervisor. No curriculum development activities 
are currently in progress to meet state competency-based 
standards. 

( 1 1 The course cun-lculum is not competency based according to 
its division supervisor. No curriculum development activities 
are currently in progre^ to meet state competency-based 
standards. 

[0] Performance data are not available. 
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Performance Indicator 12: 



The extmt that vocationai courses use computer software for skill enhancement and 
remedial education 

Performance Measure M 

The number of years that vocational courses use computer software for skill enhancement 
and remedial education 

P9rformanc9 Mmtswe Outcomes and Scores (whore 14] Is best performance) 

[A] Students in this course used computer software for skill 
OThanc^nent or remedial «lucation this year and last year, 

(3] Students In this course used computer »7ftware for skill 
enhancement or remedial education this year but not last year. 

[2] Students in this course did not use computer software for skill 
enhancefTiiNit or remedial education this year. l>rere is 
evidence that they will do so next year. 

[ 1 1 Students in this cour^ did not use computer software for skill 
enhancement or remedial education this year. There are no 
plans to have them do so next year. 

[0] Performance data are not available. 
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Information Category: Student Organization Participation 



Performance indicator 13: 

The extent of participation of students in vocational education student organizations 
Performance Measure N 

The percent of students In each vocational course participating in a vocational education 
student organization 

Porfornmnce M^asurB Outcomes and Scores {whoro (4} is best performance) 

\4] This course ranks first In its career cluster for greatest percent 
of students f^icipating In a vocational ^LK»tion student 
organization. 

[3] This couf^ ranks at>ovs the median In Its career cluster for 
greatest percent of students participating in a vocational 
education student organiaition. 

{2} This course ranks at or below the median in \ts career cluster 
for neatest perc«tt of students participating in a vocational 
education student or^nixation. 

{ 1 ] This course ranks last in Its career cluster for greatest percent 
of students f^rticipating in a vocationai education student 
CNrganization. 

(0) Performance data are not available. 
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OUTPUT COMPONENT 



Infofmation Category: Course Attrition /Completion 
Performance indicator 14: 

The extant of student attrition from vocational ^ucation vocational courses 
Performance Measure O: 

The percent of students dropping out from vocational courses during the school year 

p9rformanc9 Measure Outcomes ami Scores (where 14] is best performmce) 

{4J This course ranks among the three with the greatest percent of 
dropouui during the school year. 

13J This course ranks below the median for greatest percent of 
dropouts during the school year. 

[2} This course rarks at or at^ve the median for greatest percent 
of dropouts during the school year. 

1 1 J This course ranks among the three with the smallest percent of 
dropouts during the school year. 

{0] Performance data are not available. 

^Use the following formula to obtain the dropout percentage: the 
numbw of students who leave the course during the school 
year divided by tfie total numt^ of persons enrolled in the 
course at the beginning of the »:hooi year. 
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Peiformance tncticator 1 S: 

The extent that students complete their vocational Instruction. 
Performance Measure P: 

V\B percent that results from dividing the number of course completers by first-year course 
capacity 

Performance Measure Outcomes and Scores (whero f4J is tmat performanco) 

[4J This course rar*s among the three with ine greatest percent 
that results from dividing the number of course completers by 
first-year course capacity. 

f 3] This course ranks atK9ve the median for greatest that results 
from dividing the number of course completers by first-year 
course capacity. 

[2] This course ranks at or b9iow the median for greatest percent 
that result from dividing the number of course completers by 
first-year course capacity. 

[ 1 ] This course is among the three with the smallest percent that 
results from dividing the number of course completers by first- 
year course capacity. 

[0) Performance data are not available. 
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OUTCOME COMPONENT 



Information Category: ^o5 and oducatlon Status of Com^ leiers 
Performamse indicator 16: 

The extent of training-related job placement assistance to completers 
Performance Measure Q: 

The pBmem of completers stating that school staff placed them in triiining-related jobs 

Performance Measwe Outcmnes and Scoras ftr/iore /4/ is best performance) 

[A] This course ranks among the three with the greatest percent for 
completers stating that school staff placed them in training- 
related Jobs. 

[3] This course ranks above the median for greatest percent for 
completers stating that school staff placed them in training- 
related Jot» 

(2] This course ranks at or below the median for greatest percent 
for completers stating that school staff placed them in training- 
related |obs 

[ 1 ] This counw ranks among the thr^ with the smallest percent for 
completers stating thai srhooi staff placed them in training- 
related Ictos 

[0] Perfcmnance data are not available. 



erIc 



Performance IndlCBtiir 1 7: 

The extent that vocational course completers succeed in finding jobs or furthering their 
education 

Performance Measure R: 

The percent of vocational course completers currently In training-related jobs. In ihs m«!liafy. 
or pursuing further education 

Performance Measure Outcomes and Scores (where Ml /s bost performancs) 

{4J This course ranks among the three with the greatest percent of 
course completers either currently In tralnlng^-elated jobs. In 
the military, or pursuing further education. 

[3} This course ranks alwve the median for greatest percent of 
course completers either cun-ently In tralnli. ^related jobs, in 
the military, or pursuing further education. 

{21 This course ranks at or below the median for greatest percent 
of course completers either currently in training-related jobs. In 
t\e military, or pursuing further education. 

{ 1 } This course ranks among the three with the smallest percent of 
course completers either cun-emly In training-related jobs, In 
the military, or pursuing further education. 

10] Perfonnance data are not available. 
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Information Category: Professional Recognition 



Performance Indicator 18: 

The extent that ^nDcational course completes get licensing or certification 
Performance Measure S: 

The percent of vocational course completers who get licensed or certified 

Performance Measure Outcomes and Scores (where 14J is best performance) 

[v] This course ranks first in its career cluster for greatest percent 
of completers who get licensed or certified by a professionai 
board. 

[3] This course ranks above the median in its career cluster for 
(p-eatest percent of compietef^ who get licensed or certified by 
a professional board. 

[2] This course ranks at or below the median In its career cluster 
for greatest percent of completers who get licensed or certified 
by a professional board. 

1 1 } This course ranks last l<i Its career duster for percent of 
(x>mpieters who get license or certified by a profe^iorai 
board. 

{0] Performame data are not available. 

^Use this psrfomfiance measure indicator and outcomes only 
when comparing courses where licensing and certification are 
applicable. 
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BENEFITS COMPONENT 



Information Category: Wages 



Performance indicator 19: 

Entry-tevel wages of vocational course completers 

Performance Measure T: 

The median entry-level wages earned by vocational course completers getting training- 
related ]obs 

Performance Measure Outcomes end Scores (where [ 4] Is best performance) 

[4] This course ranks first In Its career cluster for highest median 
entry-level wage earned by completers getting training-related 
lobs. 

[31 This mjrse ranks above the median in Its career cluster for 
highest median entry-level waf;e earned by completers getting 
training-related jot». 

[2] This course ranks at or below the median in its career cluster 
for highest median entry-level wage earned by completers 
getting tralnlng^elated jobs. 

[ 1 ] This course ranks last In its career cluster for highest median 
entry-level wage earned by completers getting training-related 
jobs. 

[0] Perfonnance data are not available. 
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